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Executive Summary 

This report presents findings from a randomized control trial designed to inform the decisions of 
policymakers who are considering using online courses to provide access to Algebra I in grade 8. 

It focuses on students judged by their schools to be ready to take Algebra I in grade 8 but who 
attend schools that do not offer the course. The study tested the impact of offering an online 
Algebra I course on students’ algebra achievement at the end of grade 8 and their subsequent 
likelihood of participating in an advanced mathematics course sequence in high school. The 
study was designed to respond to both broad public interest in the deployment of online courses 
for K-12 students and to calls from policymakers to provide students with adequate pathways to 
advanced coursetaking sequences in mathematics (National Mathematics Advisory Panel 2008). 

Policymakers have persistently called for broadening access to Algebra I in grade 8. A 2008 report 
by the National Mathematics Advisory Panel recommended that “all prepared students [should] 
have access to an authentic algebra course — and [that districts] should prepare more students than at 
present to enroll in such a course by Grade 8” (2008, p. 23). 2 This recommendation echoed one 
made more than 10 years earlier by the U.S. Department of Education, which asserted that all states 
should invest in expanding access to Algebra I for middle school students (U.S. Department of 
Education 1997). 

These policy statements are built on two bodies of research. One demonstrates that Algebra I 
operates as a gateway to more advanced mathematics courses in high school and college. The other 
suggests that students who succeed in Algebra I in middle school have more success in mathematics 
throughout high school and college than students who take Algebra I later (Nord et al. 2011; Smith 
1996; Spielhagen 2006; Stevenson, Schiller, and Schneider 1994). Given these findings, federal, 
state, and local policymakers have sought to expand access to Algebra I during the past two 
decades. 

National grade 8 Algebra I enrollments increased from 16% in the 1990s to 31% in 2007 (Loveless 
2008), and in general, prior mathematics achievement is related to whether or not students take 
Algebra I in grade 8 (Walston and Carlivati McCarroll 2010). However, not all high-achieving 
students have the opportunity to take Algebra I in grade 8. An analysis of data from the Early 
Childhood Longitudinal Study (ECLS-K; U.S. Department of Education 2009a) indicated that 
nationally, approximately 25% of students who scored in the highest quartile on the grade 5 
mathematics assessment were not enrolled in a formal Algebra I course in grade 8 (Walston and 
Carlivati McCarroll 2010). 

Additional analysis of ECLS-K data (U.S. Department of Education 2009a) indicated that in rural 
schools, a larger proportion of high-achieving students do not take Algebra I in grade 8 than in 
urban or suburban schools — 39% of students who scored in the highest quartile on the grade 5 
assessment were not enrolled in Algebra I in grade 8. Furthermore, analysis of school-level 
administrator data from the ECLS-K indicated that while nationally, 16% of schools do not offer 
Algebra I to grade 8 students, the rates are highest in rural schools. About 24% of rural schools do 
not offer Algebra I in grade 8, compared with 21% of urban schools and 9% of suburban schools. 


An “authentic” algebra course is one that covers symbols and expressions, linear equations, quadratic equations, 
functions, the algebra of polynomials, and combinatorics/finite probability. 


In schools that do not offer Algebra I, curriculum offerings may be limited by constraints such as 
staffing, space, and enrollments — issues that are particularly challenging in small or rural 
schools, where student populations are low and attracting qualified and experienced teachers is 
difficult (Hammer, Hughes, McClure, Reeves, and Salgado 2005; Jimerson 2006). As technology 
capacity increases in schools around the country, online courses are increasingly seen as a viable 
means for expanding curricular offerings and broadening access to key courses, especially in 
small and rural schools (Hanum, Irvin, Banks, and Farmer 2009; Picciano and Seaman 2009; 
Schwartzbeck, Prince, Redfield, Morris, and Hammer 2003). 

This study tested the impact of expanding access to Algebra I to grade 8 students by offering an 
online course in schools that do not typically offer Algebra I in grade. It is the first randomized 
control trial testing the impact of providing an online Algebra I course on students’ mathematics 
achievement and coursetaking trajectories over time. 

Goals and Research Questions 

The primary goal of the study was to measure the effects of offering an online Algebra I course 
to algebra-ready (AR) students in grade 8 in schools that do not typically offer the course. The 
primary research questions asked whether access to online Algebra I improves AR students’ 
knowledge of algebra in the short term and whether it opens doors to more advanced 
mathematics course sequences in the longer term. The specific primary research questions were: 

1 . What is the impact of offering an online Algebra I course to AR students on their 
algebra achievement at the end of grade 8? 

2. How does offering an online Algebra I course to AR students affect their likelihood 
of participating in an advanced course sequence in high school? 

The secondary goal of the study was to estimate whether there are potential unintended 
consequences (or side effects) of offering online Algebra I to AR students. Offering the online 
Algebra I course may affect these students in unintended ways. Taking an online Algebra I 
instead of general grade 8 mathematics may, for example, adversely affect AR students’ general 
mathematics achievement. Providing online Algebra I to AR students may also have unintended 
consequences for non-algebra ready (N-AR) students — the students who remain in the general 
mathematics course. For example, when the AR students are removed from the general grade 8 
mathematics class, outcomes for the remaining students may be affected because of peer effects; 
smaller class sizes; a change in course emphasis (for example, less algebra); or other reasons. 

Four secondary research questions examined these issues: 

3. What is the effect of providing online Algebra I to AR students on their general 
mathematics achievement at the end of grade 8? 

4. What is the effect of providing online Algebra I to AR students on the algebra 
achievement of N-AR students at the end of grade 8? 

5. What is the effect of providing online Algebra I to AR students on the general 
mathematics achievement of N-AR students at the end of grade 8? 

6. How does offering an online Algebra I course to AR students affect the likelihood 
that N-AR students follow an intennediate course sequence in high school? 
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By answering the primary and secondary research questions, this study examined what happens 
to the entire population of grade 8 students — including potential benefits and possible negative 
consequences — when a school uses an online course as a way to offer Algebra I to AR students. 
The study thus sought to infonn decision makers who are considering investing in an online 
course as a means to broaden access to Algebra I in grade 8. 

The study also posed two exploratory questions that further examine the impact of online 
Algebra I on AR students’ high school coursetaking: 

7. How does access to online Algebra I in grade 8 affect the likelihood that AR students 
sign up for advanced courses in grade 9? 

8. How does access to online Algebra I in grade 8 affect the likelihood that AR students 
“double up,” or take more than one mathematics course per year, in grade 9 or 10? 

Study Design, Samples, and Measures 

This study is a randomized experimental trial. Schools in Maine and Vermont that did not 
typically offer a full section of Algebra I to grade 8 students as of the 2007/08 school year were 
eligible for the study. Initial recruitment — including determination of eligibility and interest in 
participation — focused on all public schools in Maine that serve grade 8 students (a total of 224 
schools as of fall 2007). Later recruitment focused on approximately 40 schools in Vermont that 
state administrators and local educators thought would be most likely to be eligible. A total of 68 
schools were eligible, agreed to participate in the study, and met the study’s requirement to 
identify their AR students before the schools were randomly assigned to condition. Most of these 
schools — 6 lout of 68 — were classified as rural schools by the Common Core of Data, 
maintained by the National Center for Education Statistics (http://nces.ed.gov/ccd/). In 76% of 
the schools, the grade span served was either pre-K-8 or K-8. As of the 2007/08 school year, 
grade 8 enrollments ranged from fewer than 4 students to nearly 150 students; the average was 
32 students. At the beginning of the 2008/09 academic year, 445 students had been identified the 
prior spring by their schools as AR, with an average of 6.5 AR students per school. 

Schools were randomly assigned to one of two study groups. Schools in the treatment condition 
received the online Algebra I course to offer to their AR students during the 2008/09 school year. 
Schools in the control condition did not receive the online Algebra I course during the 2008/09 
school year and implemented their usual mathematics curriculum. 

The first analytic sample included 440 AR students who attended the participating schools in 
Maine and Vermont as grade 8 students in 2008/09 (218 in treatment schools, 222 in control 
schools). At the start of the 2008/09 school year (August 2008), 445 students had been 
designated algebra ready the previous June and were enrolled in the study schools. Five of these 
445 students were excluded because their parents did not consent to their participation in the 
study. Of the 218 AR students who attended treatment schools, 211 (97%) enrolled in the online 
Algebra I course. The second analytic sample included 1,445 N-AR students (744 in treatment 
schools, 701 in control schools) who were in grade 8 in 2008/09. 

Two baseline achievement measures of general mathematics were available for all students. The 
first was the state mathematics assessment, taken by all students in the year before the study. 
These scores, collected from the states and districts, were available for more than 97% of 
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students. The second measure was a computer-adaptive pretest administered to all students in the 
study schools at the beginning of grade 8. The pretest — the Promise Assessment (Internet 
Testing Systems and SEG Assessment, 2009) — included 30 items drawn from an item bank of 
more than 1,000 items ranging in difficulty from the grade 5 to the grade 8 level. Response rates 
on the pretest exceeded 96% for both AR and N-AR students. 

The outcome measures were mathematics achievement and subsequent mathematics 
coursetaking. The mathematics achievement measures included two assessments administered at 
the end of grade 8 to all students. One assessment measured algebra knowledge; the other 
assessment measured general mathematics achievement. Each assessment was computer adaptive 
and included 20 items. The algebra assessment was scaled from 400 to 500; the general 
mathematics assessment was scaled from 200 to 400. Response rates for AR students were 99% 
on both the algebra and the general mathematics test. For N-AR students, response rates were 
86% for algebra and 93% for general mathematics. 3 4 

Research by Stevenson et al. (1994) and Schneider, Swanson, and Riegle-Crumb (1998) was 
used to code coursetaking sequences as “advanced” and “intermediate.” Data were collected at 
two time points. The first time point was the end of grade 8 (2009), when data were collected on 
students’ planned grade 9 courses (based on registrations). These data, collected to address the 
research questions on planned coursetaking, were available for 96% of the AR sample and 92% 
of the N-AR sample. For AR students, planned courses were coded as advanced if the student 
planned to enroll in a course above Algebra I in grade 9. For N-AR students, planned 
coursetaking was coded as intermediate if the student planned to enroll in Algebra I or a higher 
course in grade 9. 

The second time point was the end of grade 9 (2010), when data were collected for AR students 
only. These data included the grade 9 course names, student grades in their courses, and planned 
grade 10 courses (based on registrations). Codeable course data were collected for 97% of AR 
students. The high school course sequences were coded as advanced if the students successfully 
completed a full-year course above Algebra I in grade 9 with a grade of C or higher and enrolled 
in the next course in the sequence for grade 10. 

Description of the Intervention and Its Implementation 

The online Algebra I course used in the study is a completely web-based course offered by 
Class.com, based in Lincoln, Nebraska. The Algebra I course was one of Class. corn’s existing 
products. As implemented for the study, the online Algebra I course had three instructional 
components: the online course software, an online teacher (provided by Class.com), and an on- 
site proctor (provided by the school). 


3 The Promise Assessment pretest was administered after random assignment. Although there were no differences 
by condition observed on the prior year’s state mathematics assessment, there were differences on the pretest that 
seemed to be related to random assignment condition. Students in treatment schools outperformed students in 
control schools and spent significantly more time on the computer-based test. Because of the bias this post-random 
assignment difference could introduce into the analyses, the state mathematics assessment was used as the measure 
of baseline mathematics achievement for the main analyses. 

4 Scores based on less than five minutes of test-taking were determined to be invalid and dropped from analysis. A 
total of 1 18 algebra posttest scores were dropped from the N-AR sample for this reason (73 in the treatment group, 
45 in the control group). Multiple imputation was used to handle missing data. 
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Researchers determined that the topics covered in Class.com’ s Algebra I course were similar to 
those in typical Algebra I textbooks used in the region. The material for each topic is presented 
in the form of an electronic, interactive textbook that consists of computerized direct instruction; 
guided practice (“your-tum” problems) and practice problem sets, both with automated feedback; 
and quizzes and exams that provide immediate scores. Other activities include demonstrations of 
content materials; audio clips; interactive applets that present questions and guided solutions; a 
messaging feature through which students can send and receive messages from the online 
teachers; and a discussion board to which students can post questions and comments. 

Students taking the online course were assigned to a specific course section and taught by an 
online teacher hired, trained, and supervised by Class.com. The online teacher was responsible 
for providing instruction and supporting student learning. 

Participating schools were required to provide a school staff member to serve as an on-site 
proctor, who would supervise and support students while they were using the online course. The 
proctor did not have to be a mathematics teacher and was not required to provide instruction. 
Proctors were expected to supervise students’ behavior, serve as a contact person for students 
and parents, proctor quizzes and exams, and act as a liaison between the online teacher and the 
school, students, and parents. 

A total of 242 students enrolled in the online course (211 AR students and an additional 3 1 N- 
AR students placed into the course by their schools after random assignment). Students within 
schools were enrolled in one of 10 course sections, with an average of 24 students per section. In 
80% of treatment schools, the regular grade 8 mathematics teacher served as the proctor. In 69% 
of treatment schools, students taking the online course sat in the same classroom as students 
taking the regular grade 8 mathematics class. 

Analyses of archived data from the online course management system and data from weekly 
proctor logs showed that the types and amount of interaction between online course participants 
and their teachers and proctors did not match the initial expectations for the intervention. Online 
teachers spent less time communicating directly with students about the course than expected, 
and on-site proctors spent more time communicating with students about mathematics than 
expected. Participating students varied in the amount of the course content they completed: 43% 
of AR students completed a full Algebra I course, and 82% completed more than half the course. 

Description of the Counterfactual 

The counterfactual in the study was the “business as usual” grade 8 mathematics class taught in 
the control schools. As expected, based on review of both states’ standards for grade 8 
mathematics, students who took the general grade 8 mathematics class in control schools were 
taught algebra content. Analysis of course materials suggested that in 16 of 33 control schools 
that provided materials, the general mathematics classes had a curricular focus on algebraic 
content of 75% or higher, and 94% of the control schools had a curricular focus on algebraic 
content of 50% or higher. In more than half of control schools (55%), AR and N-AR students 
received instruction in the same mathematics classes; in 33% of control schools, AR and N-AR 
students were in separate classes (four control schools had no N-AR students). 
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Although the expected counterfactual was the absence of access to a formal Algebra I course in 
grade 8, in seven control schools, AR students took a separate Algebra I course that was taught at 
the local middle school or high school. Across all control schools, 45 AR students (20% of the 
AR control sample) took an Algebra I course in grade 8. During the recruitment process, four 
schools were deemed eligible for the study even though they allowed grade 8 students to take 
Algebra I at the local high school if they determined that students were ready and scheduling 
arrangements were possible. Nevertheless, researchers did not expect the percentage of AR 
students in control schools who took the course to be as high as it was. 

It is possible that the introduction of the study and its emphasis on Algebra I for grade 8 students 
into these schools changed the nature of offerings to students in control schools. However, 
without another comparison group of schools that was completely unaffected by the study, there 
is no way to know whether this assertion is true. 

Analytic Approach 

The analyses for this study compare outcomes for students in treatment schools with their 
counterparts in control schools at the end of grade 8 (spring 2009) and, for AR students only, at 
the end of grade 9 (spring 2010). All analyses of outcomes at the end of grade 8 were conducted 
separately for the AR and N-AR student samples; AR student outcomes were never compared 
with N-AR student outcomes. 

Analyses of the algebra and general mathematics posttests used hierarchical linear modeling 
(Raudenbush and Bryk 2002), accounting for the nesting of students within schools and 
controlling for student- and school-level covariates. Results are reported both in their original 
metric and as effect sizes. The analyses of coursetaking sequences used hierarchical generalized 
linear models that assume a Bernoulli sampling distribution and logit link function (Raudenbush 
and Bryk 2002; McCullagh and Nelder 1998). These models, appropriate for use with binary 
outcomes, accounted for nesting of students within schools and included the same student- and 
school-level covariates as the models used for the achievement outcome measures. 

Results 

The results indicate that offering Algebra I as an online course to AR students is an effective way 
to broaden access in schools that do not typically offer Algebra I in grade 8. Taking this course 
significantly affected students’ algebra achievement at the end of grade 8 and increased their 
likelihood of participating in an advanced coursetaking sequence in high school. The course had 
no discernible side effects on AR students’ general mathematics achievement at the end of grade 
8 or any negative effects on N-AR students’ measured outcomes. 

Primary Analyses 

The results of the primary analyses indicated the following: 

• Access to online Algebra I in grade 8 had a positive impact on AR students’ algebra 
achievement at the end of grade 8. AR students in treatment schools outperformed those 
in control schools by approximately 5.5 scale score points (model-adjusted mean scores 
were 447.17 for the treatment group and 441.64 for the control group; p = 0.001; effect 
size = 0.39). 
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• Access to online Algebra I in grade 8 had a positive impact on AR students’ high school 
coursetaking: AR students from treatment schools were significantly more likely to 
follow an advanced course sequence in high school than their peers from control schools. 
The average (model-predicted) probability of participating in an advanced course 
sequence was 0.5 1 for AR students from treatment schools and 0.26 for AR students from 
control schools — a difference of 0.25 (p = 0.007). These results mean that AR students 
from treatment schools were twice as likely to participate in an advanced course sequence 
than AR students from control schools. 

To examine the robustness of these findings, researchers estimated models that examined 
different specifications or baseline covariates. The results of all of the sensitivity analyses were 
consistent with the main analyses reported. However, for both outcomes (algebra scores at the 
end of grade 8 and high school coursetaking), the magnitude of the impact estimate was 
attenuated when the Promise Assessment general mathematics pretest administered for the study 
was used as the baseline measure of prior achievement instead of the state mathematics 
assessment scores. For algebra scores at the end of grade 8, the effect of online Algebra I was 
still statistically significant (effect size = 0.29, p = 0.009); for high school coursetaking, the 
difference in the probability of participating in an advanced course sequence between treatment 
and control students fell below the level of statistical significance (p = 0.037). 5 

Secondary Analyses 

The results of the secondary analyses indicated the following: 

• AR students’ general mathematics achievement at the end of grade 8 did not appear to be 
affected by the online Algebra I course. Model-adjusted mean scores were 361.42 for the 
treatment group and 357.82 for the control group (p = 0.204; effect size = 0.14). 

• N-AR students did not appear to be affected by their schools’ adoption of the online 
Algebra I course (offered to their AR peers) on any of the measured outcomes (algebra 
posttest: p = 0.443, effect size = 0.06; general mathematics posttest: p = 0.789, effect size 
= 0.02; planned grade 9 coursetaking: p = .099, difference in predicted probability = 0.10, 
in favor of students from treatment schools). 

All sensitivity analyses testing the robustness of the secondary analyses confirmed the results 
reported in the main analysis, with one exception. In Model E.22, which used complete case 
analysis instead of multiple imputation for missing data, the difference in predicted probability of 
enrolling in an intermediate course sequence reached the level of statistical significance, 
indicating that N-AR students in treatment schools were more likely than those in control 
schools to enroll in an intermediate course sequence in grade 9. Nevertheless, all of the main and 
sensitivity analyses indicated that the offering of the online course to AR students did not have 
deleterious effects on AR students’ general mathematics scores or on any of the measured N-AR 
student achievement and coursetaking outcomes. 


5 The coursetaking result is not statistically significant after adjusting the alpha level for multiple comparisons for 
the two primary outcomes. The adjusted alpha level applied to both primary analyses was 0.025. 



Exploratory Analyses 

Two sets of exploratory analyses examined AR students’ high school coursetaking more closely. 
First, researchers examined the effect of online Algebra I in grade 8 on the likelihood of planned 
enrollment in an advanced grade 9 course, as of the end of grade 8. Like the main analyses, 
which focused on coursetaking after a full year of high school, this exploratory analysis showed 
that, at the end of grade 8, AR students from treatment schools were 3.38 times more likely to 
enroll in an advanced course (that is, a course above Algebra I) than were AR students from 
control schools (p = 0.005). 

Second, researchers used the follow-up data collected from high schools at the end of grade 9 to 
examine the course progressions through which AR students either met or did not meet the 
criteria for an advanced course sequence. The evidence indicated that 21% of the AR sample (96 
students) took more than one full-year mathematics course in grade 9, planned to do so in grade 
10, or both. 

An exploratory impact analysis tested the effect of access to online Algebra I in grade 8 on the 
likelihood that AR students “doubled up” (that is, took more than one mathematics course at a 
time) in grade 9 or 10. The results indicated that students in control schools (predicted 
probability = 0.23) were more than twice as likely to double up in mathematics courses in grade 
9 or grade 10 than AR students in treatment schools (predicted probability = 0.10) (p = 0.033). 
This suggests that in addition to affecting whether students pursue an advanced sequence, the 
intervention had an impact on how students enter an advanced course pathway. High-achieving 
students without access to Algebra I in grade 8 may get themselves on track for an advanced 
course sequence by taking two mathematics courses at a time. The study results show that access 
to online Algebra I in grade 8 can reduce the need to double up, allowing students who would 
otherwise have done so to take other high school courses and focus on one mathematics content 
area at a time. 

Limitations of the Study and Future Research Directions 

This study was conducted with a sample of schools in Maine and Vennont that met the eligibility 
criteria for participation and agreed to take part in a random assignment study. Many of these 
schools were small (48% had grade 8 enrollments of less than 17 students), and 90% were in 
rural areas. Analyses of longitudinal data from the ECLS-K study (U.S. Department of Education 
2009) indicate that a significant proportion of schools do not offer Algebra I to grade 8 students 
(approximately 16% nationally), and moreover, that the proportion of schools in rural areas with 
limited access to Algebra I is higher than in urban and suburban areas. Still, it is not clear 
whether the study schools represent small rural schools located in other parts of the region or 
country or the extent to which the results observed in these schools generalize to other schools 
interested in using online courses to expand access to Algebra I to grade 8 students. 

Although the consent rates and response rates were high (above 95% in the AR sample and 
above 85% in the N-AR sample), they were not 100%. Multiple imputation was used to adjust 
for any bias nonresponse might introduce, but it is not impossible that bias was nonetheless 
present. 
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The online course chosen, Class.com’ s Algebra I course, is similar in content and focus to the 
offerings of other providers. However, it is not clear that similar results would have been 
observed had another course provider been chosen. Moreover, the results observed in this study 
cannot necessarily be generalized to more recently developed online courses. 

For all these reasons, replication of this study is necessary to gain a better understanding of the 
potential impacts of using an online course to expand access to Algebra I to grade 8 students. In 
particular, future studies should examine longer-term effects of access to online Algebra I in 
grade 8 — through high school, college, and even beyond. This study included a one-year follow- 
up to track students from grade 8 into high school. A longer study is needed to assess whether 
access to online Algebra I in grade 8 continues to impact participation in advanced mathematics 
course sequences through the end of high school. 

As the use of online courses continues to rise in U.S. schools, future research should continue to 
study their short- and long-term effects on student coursetaking patterns and achievement in key 
content areas. Further investigation of the effectiveness of online courses should contrast the 
offering of them with various relevant business-as-usual situations. These include school settings 
where students’ lack of access to specific courses (where the control group does not take the 
course) as well as school settings where particular courses are oversubscribed or taught by under- 
qualified or uncertified teachers (where the control group would take a standard face-to-face 
version of the online course). 

Schools around the country, particularly those in rural areas, are in search of innovative ways to 
expand their course offerings. To address this need, this study focused on the use of an online 
course to provide access to Algebra I in schools that do not typically offer the course in grade 8. 

It did not compare the effects of taking online Algebra I versus a standard face-to-face version of 
the course in grade 8, and the results should not be interpreted to indicate that offering online 
Algebra I is better than (or as good as) offering a face-to-face Algebra I course to eighth graders. 
In addition, given that the study compared the offering of an online Algebra I course to a lack of 
(or limited) access to Algebra I in grade 8, it is not possible to isolate the portion of the observed 
effects that is due to the fact that the course was online. The content of the course (Algebra I) 
cannot be untangled from the mode of instruction (online). Thus it is possible that broadening 
access to any type of formal Algebra I course to AR grade 8 students would yield similar effects. 

Conclusions 

This study is the first of its kind to rigorously evaluate the impact of offering an online version of 
Algebra I in schools that otherwise do not typically offer the course, even though they have 
students who are ready to take it. For educators and students facing similar challenges, the results 
of this study may be particularly informative and promising. Results showed that offering an 
online course to AR students is an effective way to broaden access to Algebra I in grade 8 and 
later, to more challenging mathematics course opportunities. The study demonstrates that an 
online course as implemented is more effective in promoting students’ success in mathematics 
than existing practices in these schools. 
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Chapter 1 Introduction and Study Overview 

A 2008 report by the National Mathematics Advisory Panel recommended that “all prepared 
students [should] have access to an authentic algebra course — and [that districts] should prepare 
more students than at present to enroll in such a course by Grade 8” (National Mathematics 
Advisory Panel 2008, p. 23). 6 This recommendation echoed one made by the U.S. Department of 
Education (1997), which asserted that all states should invest in expanding access to Algebra I 
for middle school students. 

Both of these policy statements are built on mainly correlational research showing that Algebra I 
operates as a gateway to more advanced mathematics courses in high school and college and that 
students who succeed in Algebra I in middle school have more success in mathematics 
throughout high school and college. For example, Stevenson, Schiller, and Schneider (1994) 
found that grade 8 students who take Algebra I are more likely than their peers to take advanced 
mathematics courses in high school. Smith (1996), using a large nationally representative dataset 
and controlling for student background characteristics, found that providing access to Algebra I 
in grade 8 has a sustained, positive relationship with students’ high school mathematics 
attainment and achievement. Spielhagen (2006) concluded that students who take Algebra I in 
grade 8 take more advanced mathematics courses in high school and attend college at higher 
rates than students with similar academic backgrounds who take Algebra I in high school. More 
recent analyses of high school transcript data indicate that almost two-thirds of students who 
took Pre-calculus or higher by the time they graduated from high school in 2009 had taken 
Algebra I by the eighth grade (Nord et al. 201 1). 

The federal policy recommendations and state and district initiatives that followed the earlier 
studies may have affected grade 8 Algebra I enrollments throughout the country over the past 
two decades. In the 1990s, 16% of grade 8 students were enrolled in Algebra I nationwide; by 
2007 this figure had risen to 31% (Loveless 2008). A U.S. Department of Education report on 
grade 8 algebra, using data from the Early Childhood Longitudinal Study (ECLS-K), reports 
similar statistics. In the 2006/07 school year, 33% of grade 8 students in the national sample 
were enrolled in Algebra I (another 6% were enrolled in courses above Algebra I); in general, 
students with higher levels of prior achievement (as measured by the ECLS-K mathematics 
achievement test taken at the end of grade 5) were more likely to be enrolled in Algebra I in 
grade 8 (Walston and Carlivati McCarroll 2010). 

Despite this increased enrollment trend, not all high-achieving students have the opportunity to 
take Algebra I in grade 8. Walston and Carlivati McCarroll (2010) found that approximately 
25% of students who scored in the highest quartile on the ECLS-K grade 5 mathematics 
assessment were not enrolled in a formal Algebra I course in grade 8. 

Additional analysis of ECLS-K data indicated that this finding is particularly pronounced in rural 
schools. In rural schools, 39% of high-achieving students (that is, students who scored in the top 
quartile in grade 5) did not take Algebra I (or higher) in grade 8. This is significantly higher than 
the proportion of high-achieving students in urban or suburban schools who did not take the 


6 An “authentic” algebra course is one that covers symbols and expressions, linear equations, quadratic equations, 
functions, the algebra of polynomials, and combinatorics/finite probability. 
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course (26.4% and 26.3%, respectively). 7 Furthermore, an analysis of school-level administrator 
data from the ECLS-K indicates that while nationally, 16% of schools do not offer Algebra I to 
grade 8 students, the rates are highest in rural schools. About 24% of rural schools do not offer 
Algebra I in grade 8, compared with 21% of urban schools and 9% of suburban schools. 8 

In such schools, curriculum offerings may be limited by constraints such as staffing, space, and 
enrollments — issues that are particularly challenging in small or rural schools, where student 
populations are low and attracting qualified and experienced teachers is difficult (Hammer et al. 
2005; Jimerson 2006). As technology capacity increases in schools around the country, online 
courses are increasingly seen as a viable means for expanding curricular offerings and 
broadening access to key courses, especially small and rural schools (Hanum et al. 2009; 
Picciano and Seaman 2009; Schwartzbeck et al. 2003). 

This study was conducted by the Regional Educational Laboratory — Northeast and Islands 
(REL-NEI). It focused on broadening access by offering an online course to grade 8 students 
considered by their schools to be ready for a formal Algebra I course but who would not 
typically take one until high school because they attend middle schools that do not offer the 
course. 

The primary goal of this study was to determine whether broadening access to Algebra I in grade 
8 by offering an online course improves students’ knowledge of algebra in the short term, opens 
doors to more advanced course sequences in the longer term, or both. 

The secondary goal was to estimate potential unintended consequences (side effects) of offering 
online Algebra I to students considered algebra ready by their schools. Offering an online 
Algebra I course may affect students who take the course in unintended ways — by, for example, 
affecting their general mathematics achievement. Offering an online course may also have 
unintended consequences on the grade 8 students in the school who remain in the general 
mathematics course — through, for example, peer effects, changes in course emphasis, or smaller 
class sizes. 

This study was designed to assess the effects — positive and negative — on all grade 8 students of 
offering an online Algebra I course. It sought to produce useful information for education 
decision makers considering investing in an online course as a means of broadening access to 
Algebra I in grade 8. 


7 REL-NEI researchers conducted this analysis with the child-level ECLS-K K-8 Full Sample Public-use Data File 
(U.S. Department of Education 2009a). The ECLS-K study collected data from 8,703 students when they were in 
grade 8. After classifying the students as high achievers (based on their grade 5 assessment scores) and whether (or 
not) they reported taking a grade 8 mathematics course at or above Algebra I, the researchers used a Chi-square test 
to examine whether the proportion of high-achievers who did not take Algebra I (or higher) was different by locale 
(small town/rural vs. large/mid-sized city vs. suburban/large town). The difference in proportions was statistically 
significant,/ 2 = 19.4, df= 2,p< 0.001. 

s Administrators of schools attended by ECLS-K student participants reported the proportion of their grade 8 
students taking different mathematics courses. These data are available for 1,946 public schools. REL-NEI 
researchers classified schools as either offering Algebra I to grade 8 students or not, and used a Chi-square test to 
examine whether the proportion of schools that did not offer Algebra I differed by locale. The difference was 
statistically significant,/ 2 = 49.3, df= 2 ,p< 0.001. 
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Significance of Algebra I 

Algebra I is a gatekeeper course because it is a prerequisite for the high school mathematics and 
science courses considered essential, if not required, for getting into college. High school 
mathematics courses are ordered sequentially; students must successfully complete Algebra I 
before taking subsequent mathematics courses (Smith 1996; Wagner and Kieran 1989). If 
students succeed in Algebra I, they typically take Geometry, Algebra II, and then more advanced 
courses, such as Trigonometry, Pre-calculus, and Calculus. 9 Several research studies have shown 
that success in Algebra I is highly correlated with enrollment in more advanced mathematics and 
science courses (Lacampagne, Blair, and Kaput 1995; Atanda 1999; Kilpatrick, Swoffard, and 
Findell 2001; Nord et. al, 2011). 

Previous research, mainly correlational, suggests that having access to Algebra I in grade 8 
benefits at least some students. Using data from the 1988 National Educational Longitudinal 
Study, Stevenson, Schiller, and Schneider (1994) examined the relationship between students’ 
mathematics and science opportunities in grade 8 and their later opportunities in mathematics 
and science in high school (see also Schneider, Swanson, and Riegle-Crumb 1998). They defined 
three course sequences: 

• Sequence A (advanced): Completion of both Geometry and Algebra II or any higher level 
course by grade 10 

• Sequence B (intermediate): Completion of either Geometry or Algebra II by grade 10 

• Sequence C (low): Completion of neither Geometry nor Algebra II by grade 10. 

According to the study, 42% of students who took Algebra I in grade 8 participated in an 
advanced course sequence in high school. In contrast, only 12% of students who did not take 
Algebra I in grade 8 participated in an advanced course sequence. The study’s authors conclude 
that mathematics course opportunities in grade 8 are related to students’ subsequent opportunities 
to take, and succeed in, advanced course sequences in high school. Another study using the same 
data found that 60% of students who took Calculus by grade 12 had taken algebra in grade 8 
(U.S. Department of Education 1996). 

Research also suggests that students who take Algebra I in middle school subsequently have 
higher mathematical skills that are sustained over time. Smith (1996) used data from the High 
School and Beyond study to estimate the relationship between middle school algebra and later 
mathematics outcomes, controlling for differences in student background (social and 
demographic background, aptitude, and academic emphasis or interest in mathematics). Her 
results suggest that early access to algebra is related to high school mathematics coursetaking 
behavior and achievement above and beyond these observable or measurable characteristics. 


9 The Algebra I -> Geometry A Algebra II sequence is known as the traditional mathematics coursetaking pathway 
(National Mathematics Advisory Panel 2008; Common Core Standards Initiative 2010). Some schools reverse the 
order of Algebra II and Geometry, yielding an Algebra I -> Algebra II A Geometry sequence; this ordering is less 
common than the traditional sequence. Other schools offer an integrated course pathway that combines the content 
of Algebra I, Geometry, and Algebra 11 into integrated courses, which typically have generic names, such as 
Mathematics 1, Mathematics 2, and Mathematics 3 (National Mathematics Advisory Panel 2008; Common Core 
Standards Initiative 2010). 
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Students who took Algebra I in middle school completed an average of one more year of 
mathematics courses than students who took Algebra I in high school (2.3 versus 1.3 years). 

They also outscored their counterparts who took Algebra I in high school on the High School and 
Beyond mathematics assessment in both grades 10 and 12. The differences in these outcomes 
were statistically significant (p < 0.001), controlling for students’ background characteristics 
(Smith 1996). 

Spielhagen (2006) used data from a large urban district to compare high school and college 
outcomes of students with and without access to Algebra I in grade 8. She concluded that grade 8 
students with similar academic abilities who were provided access to Algebra I followed a more 
advanced coursetaking sequence in high school than students who took Algebra I in grade 9. 
Specifically, 82% of students who completed Algebra I in grade 8 were enrolled in a course 
above Algebra II in grade 11, compared with just 2% of their peers who did not complete 
Algebra I in grade 8. Students who completed Algebra I in grade 8 were also more likely to 
attend college (62%) than students who did not have access to Algebra I in grade 8 (31%). 

Other research indicates that advanced coursetaking in high school (typically defined as 
completing courses above Algebra II) is related to the likelihood of enrolling in and completing 
college. Horn and Nunez (2000) found that three-quarters of students who participated in an 
advanced coursetaking sequence in high school enrolled in four-year colleges. Adehnan (1999, 
2006) found that the odds of completing college are twice as high for students who take a 
sequence of advanced mathematics courses in high school. Enrollment in higher-level 
mathematics and science courses is also related to future educational and employment 
opportunities (Gamoran and Hannigan 2000; U.S. Department of Education 1997). Rose and 
Betts (2001) showed that students who take higher-level mathematics classes in high school have 
higher earnings 10 years after high school graduation, even after controlling for background 
characteristics and eventual educational attainment (see also Jabon et al. 2010). 

Policy statements emphasizing the importance of offering Algebra I in middle school (for 
example, U.S. Department of Education 1997; National Council of Teachers of Mathematics 
2000) have suggested that improving student performance in mathematics in high school and 
college requires ensuring that more students are prepared during their middle grade years to 
move successfully through upper-level mathematics courses in high school. In addition to 
expanding access to a fonnal Algebra I course in grade 8, these policy recommendations have 
likely influenced the degree to which algebraic concepts are infused into general middle school 
mathematics curricula and standards. For example, the National Council of Teachers of 
Mathematics has an algebra strand in its middle school content standards, and these standards are 
often reflected in middle school mathematics textbooks and state standards (National Council of 
Teachers of Mathematics 2000, 2006). 

It is important to note that the research documenting the relationships between taking Algebra I 
in grade 8 and high school coursetaking and between high school coursetaking and 
postsecondary outcomes is mainly correlational. As such, it is subject to problems of self- and 
school selection of students into the course and subsequently into more advanced courses 
(Loveless 2008). None of the research studies to date used a random assignment design to 
establish statistically equivalent groups of students who do and do not take Algebra I in grade 8. 
This study, therefore, is the first to rigorously test the impact of providing access to Algebra I in 


4 



grade 8, to students considered ready for the course. More specifically, and as described in the 
following sections, the study focused on providing access by offering an online course. 

The Use of Online Courses to Broaden Access 

In schools that do not offer particular classes because of a lack of resources such as space and 
available teachers, online courses are one way to provide courses to interested or eligible 
students. Offering coursework virtually is a strategy that schools use to expand the curricula 
available to their students (National Education Association 2006), particularly in small schools 
and isolated communities that do not have access to critical courses in science, technology, 
engineering, and mathematics (Picciano and Seaman 2009; Tucker 2007). 

The increasing popularity of online courses is driven by both technological advancements and 
the flexibility with which online courses can provide access to content and instruction (U.S. 
Department of Education 2009). Online courses allow schools to take advantage of a broader 
pool of qualified teachers, which can enable students to take courses that are otherwise not 
offered or taught by qualified teachers. 

The use of online courses in K-12 settings has been on the rise over the past decade. According 
to the National Center for Education Statistics, 37% of school districts used technology-based 
distance learning during 2004/05 (Zandberg and Lewis 2008). In 2007/08, a national survey of 
K-12 public school districts conducted by the Sloan Consortium found that 75% of school 
districts had one or more students enrolled in online courses and that the total number of K-12 
public school students engaged in online courses exceeded 1 million — a 43% increase from the 
700,000 students reported in 2005/06 (Picciano and Seaman 2007, 2009). As of 2007, 28 states 
had virtual high school programs, enabling students to take online courses in addition to their 
school-based courses to fill curriculum gaps (for example, Advanced Placement [AP] courses) or 
providing opportunities for credit recovery (Tucker 2007). 

Surveys of K-12 public schools have suggested that rural districts and schools are especially 
interested in online learning. In the Sloan Consortium surveys, respondents from small rural 
school districts reported that they use online courses to provide opportunities they would not 
otherwise be able to offer (Picciano and Seaman 2009). 

Two other surveys examined the prevalence of rural schools’ use of distance learning, a broader 
category that overlaps with online learning. Among the 896 superintendents of rural school 
districts interviewed by Schwartzbeck et al. (2003), 62% used distance learning to offer 
additional courses, with mathematics courses making up 22% of their distance learning offerings. 
In another nationally representative sample of 417 rural school districts, 85% reported having 
used some form of distance education at some time, with 69% reporting that they were currently 
doing so; of those currently using distance education, 89% reported that they needed distance 
education to offer advanced or enrichment courses (Hanum et al. 2009). 

Of the 68 schools participating in this study, 61 are rural, as defined by the Common Core of 
Data. According to a 2007 report by the Rural School and Community Trust Policy Program, 
Maine has the highest percentage of students attending rural schools of any state in the country 
(52.9%), and Vermont has the second-highest percentage (51.3%) (Johnson and Strange 2007). 
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Thus, this study context was an ideal setting for examining the effectiveness of using an online 
course to expand access to Algebra I for grade 8 students. 

Research on the Effectiveness of Online Courses 

No rigorous studies have tested the efficacy of using online courses to broaden access to courses 
students could not otherwise take. Studies of virtual course programs show promising 
completion rates and satisfaction among students, teachers, and administrators (Optimal 
Performance 2006), but studies have not indicated whether online courses benefit students who 
would not otherwise have had access to such course subjects. This study addresses this gap. 

A small body of literature compares online learning with traditional face-to-face learning, 
particularly at the postsecondary level. Though this study was not designed to compare online 
Algebra I to face-to-face Algebra I, this literature is relevant to this study because it provides 
information on the utility of online courses as an educational experience compared with 
traditional face-to-face coursework. A meta-analysis of 99 studies that met the authors’ criteria 
for rigor finds that online instruction yields positive effects relative to face-to-face instruction 
(U.S. Department of Education 2009b). 10 This finding is based almost entirely on postsecondary 
education. 

Just 5 of the 99 studies included in the meta-analysis were conducted at the K-12 level, 
indicating the need for rigorous research on K-12 online learning. One of these studies 
(O’Dwyer, Carey, and Kleiman 2007) is particularly relevant, because it examined the effects of 
an online Algebra I course. In this quasi-experimental study, grades 8 and 9 students in 18 
classrooms participating in the Louisiana Algebra I Online Project were compared with students 
in comparison classrooms on outcomes including an end-of-year Algebra I assessment. The 
authors compared pretest scores on a general mathematics assessment and demonstrated that the 
online and comparison students were not statistically different at baseline. At the end of the 
school year, the difference in scores between students in the two types of classrooms on an 
algebra posttest was not statistically significant (average scores on the 2 5 -point test were 15.27 
for the online group and 14.61 for the comparison group,/? = 0.093). These findings provide 
suggestive evidence that online Algebra I can produce outcomes that are similar to the outcomes 
in traditional Algebra I courses. It is possible that 18 classes per group was too small a sample 
size to detect a true effect on the algebra posttest. Students in the online class reported feeling 
less confident in their algebra skills, but they outscored students in comparison classrooms on 
groups of items that required creating an algebraic expression from a real-world example. This 
study provides foundational information on potential effects of online Algebra I. The current 
study, while neither designed nor intended to compare the effectiveness of online vs. face-to-face 
Algebra I courses, builds on these findings to investigate the impact that access to an online 
Algebra I course has on academic outcomes for both course participants and their 
nonparticipating peers. 


10 Studies that qualified for inclusion in the meta-analysis used an experimental or quasi-experimental design (if 
quasi-experimental, the study must have included statistical controls for prior achievement). They also reported data 
sufficient for calculating effect sizes per the What Works Clearinghouse (2007) guidelines. 
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Selection of the Online Course for This Study 

The goal of this study was to examine the impact of offering an online Algebra I course to 
broaden access to grade 8 students, not to study the effectiveness of a particular course 
provider’s online Algebra I course. In fall 2007, REL-NEI established the following criteria for 
selecting an online Algebra I course provider: 

• The content of the Algebra I course represents what is typically part of a high school 
level Algebra I course. 11 

• The course provider is established and has a widespread presence in secondary 
schools. 

• The course provider offers courses that are geared toward a range of learners, 
including courses used for both remediation and acceleration (that is, it does not 
specialize only in AP courses). 

• The course provider is willing to offer individual seats in a course for as few as one 
student per study school (at a competitive per student fee). 

• The course provider hires, trains, and supervises the online teachers for the purpose of 
the study. (This criterion was considered essential given the study’s focus on rural 
schools, which typically do not have the resources to provide an Algebra I teacher or 
course to their grade 8 students.) 

• The provider is willing to participate in a randomized study and operate within the 

12 

study’s parameters. 

The study team identified potential course providers by conducting Internet searches, reviewing 
the providers listed in the National Repository of Online Courses, and contacting course 

• IT.. 

providers known to REL-NEI from previous research projects. It identified 1 1 online course 
providers that offered an Algebra I course. After reviewing the online course providers, REL- 
NEI determined that all 1 1 providers met the first two criteria and reached out to all of them. 
REL-NEI contacted each organization at least three times to request a conversation. Four of the 
1 1 providers indicated initial interest. One of these providers did not want to be included in an 
evaluation study, one indicated that its current focus was to provide Advanced Placement 
courses, and a third was no longer interested once it understood the scope of the study. At the 
conclusion of the interviews, only the online Algebra I course provided by Class.com met all the 
criteria required for the study. 


1 1 See appendix C for a comparison of the learning objectives and course content of the Class.com Algebra I course 
and standard Algebra I courses. 

12 For example, the provider needed to be able to implement the program in the fall without knowing the number of 
students who would be enrolled until the summer before implementation and to structure online sections that 
encompassed a variety of schools, schedules, and calendars. 

13 Terms used in the internet search included “online course providers,” “online courses,” “algebra online,” “algebra 
AND online,” “virtual school,” “virtual school algebra,” “algebra I online,” “distance education,” and “distance 
education AND algebra.” 
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Overview of the Class.com Online Algebra I Course 

The Class.com course content was originally developed by teams from the University of 
Nebraska, funded by a 1996 U.S. Department of Education Star Schools grant. Class.com was 
created to make these courses available on an ongoing basis. It has been a self-sustaining 
business since 1999. Class.com is a privately held small business that partners with more than 
4,400 secondary schools across the country. At the time of selection, its courses were determined 
to meet the guidelines of the Southern Regional Education Board’s Essential Elements for Web- 
Based Courses for High School Studen ts (Thomas 1999), which was widely used as the 
benchmark for quality assurance by states for online education. 14 

Class.com had not delivered its Algebra I course to middle school students before the study. It 
assumed that AR students would be ready for a high school version of Algebra I and therefore 
did not reduce the academic rigor for delivery in grade 8. 

Like online classes offered by other providers, Class.com classes contain many of the same 
instructional components as a traditional face-to-face class, including defined learning 
objectives, curriculum materials, assignments, problem sets, quizzes and tests, and grades. The 
online Algebra I course used in this study provides a structure and a schedule of requirements 
throughout the school year. Like traditional courses, courses are structured into major topic units 
and individual lessons within units. Students are expected to follow a schedule in completing 
each lesson and to be prepared for quizzes and tests as scheduled. 

Online and traditional classes have many parallels but also some critical differences, particularly 
in the forms of communication available to students and teachers. In the Class.com online 
Algebra I course, the primary form of communication is asynchronous online exchanges (that is, 
the student and the teacher are not online at the same time). Because the interactions are online, 
class participants do not need to gather in one place and can be distributed in different locations. 
Because the interactions are asynchronous, class participants within the same section do not need 
to be online at the same time. 

Intended Implementation of the Online Course in This Study 

For AR students in treatment schools, the online Algebra I course was offered instead of regular 
mathematics instruction. As in a traditional class, students taking the online course were assigned 
to a specific section and taught by a teacher who was responsible for providing instruction and 
supporting their learning. Class.com recruited teachers to teach the online Algebra I course in 
participating schools. Their responsibilities included grading homework, monitoring student 
understanding, promoting student engagement, conducting online discussions, and monitoring 
student performance on quizzes and exams. This online course model was intended to be 
different from online tutor approaches, in which the primary interactions are between individual 
students and the computer. 


14 These standards include providing instruction that engages students in activities that address various learning 
styles; opportunities for students to engage in abstract thinking; appropriate teacher-to-student interactions, such as 
feedback on student progress; opportunities for appropriate interaction among students; and assessment of students’ 
content mastery using fair, adequate, and appropriate methods and procedures. 
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All schools were required to specify a class period for students take the course. However, the 
specified time was allowed to differ for students in the same class, students from different 
schools could be in the same online section, and students could access the class website at any 
time, in addition to the scheduled class period (although out-of-school access to the Internet was 
not a requirement for participating in the intervention). 

All participating schools were required to provide an on-site proctor to monitor students while 
they were accessing the course. In addition to possessing basic computer skills, the proctor was 
expected to supervise students’ behavior, serve as a contact for students and parents, proctor 
quizzes and exams, and act as a liaison between the online teacher and the school, students, and 
parents. The proctor was not required to provide instruction and therefore did not need to be a 
mathematics teacher. 

Because teachers and students typically were not online at the same time, they communicated 
through an online course messaging system or discussion boards. 15 Online teachers were 
encouraged to send students personal messages regarding each day’s work or homework 
submitted the previous day and to reply to any student messages within 24 hours. (Chapter 3 
provides more information on the intended implementation of the online course.) 

Programmatic Features of the Intervention 

The online Algebra I course is a multidimensional intervention with programmatic features that 
make it different from “business as usual” mathematics instruction. In addition to changing the 
instruction received by students taking Algebra I, offering an online algebra course may change 
the grade 8 mathematics program available to students not taking Algebra I. Six programmatic 
features of the online Algebra I course may cause changes in grade 8 mathematics: 

• Course content : AR students in schools that offer the online course have access to a 
course with typical Algebra I content. In the absence of the course, AR students take a 
general grade 8 mathematics course whose content includes algebraic concepts but is not 
a formal Algebra I course. 16 

• Mode of instruction: The online Algebra I course is completely web based and has three 
instructional components: the online course software, the online teacher, and the on-site 
proctor. Regular grade 8 mathematics classes may use technology to varying degrees, but 
the classes are not online courses and instruction is delivered primarily face to face. 

• Teacher qualifications'. Online teachers are required to be certified and trained in both the 
content and the delivery mode. Not all teachers in the regular mathematics course may be 
certified. 

• Staffing intensity'. Adoption of the online course adds a teacher — the online teacher. 
Schools using the online course were also required to assign a proctor to support students 


15 Students and online teachers did not use email to communicate. All communications among teachers and students 
occurred within the learning management system. 

16 A review of the mathematics content standards in both states before this study was launched revealed that all 
middle school students are expected to learn algebraic content in grade 8. 
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taking the course. In contrast, the regular grade 8 mathematics program is staffed by the 
usual number of educators in schools that do not offer online Algebra I. 

• Class size : In schools that offer the online Algebra I course, the AR students are removed 
from the general grade 8 mathematics class to take the online course. Therefore, N-AR 
students are in general mathematics classes that are smaller than they would have been 
without the online Algebra I course. 

• Ability grouping: In schools that offer the online Algebra I course, higher-ability students 
are removed from the general grade 8 mathematics class to take the course. As a result, 
the remaining students are more homogenous and lower in average ability than they 
would have been had the online course not been offered. This change in the composition 
of the general mathematics class may result in changes in the content taught. For 
example, the course may be less challenging than it would have been or place less 
emphasis on algebra concepts than it would have had the AR students remained in the 
class. In the absence of the intervention, students are not necessarily separated into ability 
groups for mathematics instruction. 

Given the multidimensional nature of the intervention and the research design, it is impossible to 
attribute impacts to these six features individually. Nevertheless, it is important to identify these 
features to understand the comprehensive effects that adopting such a course can have. In 
Chapter 3, the implementation of mathematics instruction in both treatment and control schools 
is described in relation to these six dimensions. 

Study Purpose and Research Questions 

The goal of this randomized controlled trial is to ascertain the effects of broadening access to 
Algebra I to grade 8 students using an online Algebra I course in schools that do not typically 
offer Algebra I. In particular, this study addresses the following policy question: 

In schools that do not offer Algebra I to grade 8 students, what are the effects of 
offering Algebra I as an online course to students who are considered algebra ready? 

This policy question concerns the effects on student achievement in mathematics and subsequent 
high school mathematics coursetaking for all grade 8 students in a school. 

The primary focus was on the effects on AR students. Students were identified as AR before 
random assignment to ensure the comparability of students in treatment and control schools. The 
online course was then offered to AR students in the treatment schools. 

The secondary goal of the study was to determine whether the delivery of the online course has 
side effects on AR students’ general mathematics achievement or on outcomes for N-AR 
students in participating schools. Implementing the online Algebra I course required removing 
AR students from the regular grade 8 mathematics class. This removal may impact AR students’ 
general mathematics scores — an effect that is important to teachers and administrators concerned 
with their students’ individual achievement as well as their school’s performance on grade 8 state 
mathematics assessments. The removal of AR students may also have indirect effects on N-AR 
students as a result of the change in class size and composition. 


10 


Primary Research Questions 

The two primary research questions focus on the direct effects of broadening access to Algebra I 
by offering an online version of the course to AR students: 

1 . What is the impact of offering Algebra I online to AR students on their algebra 
achievement at the end of grade 8? 

2. How does offering Algebra I online to AR students affect their likelihood of 
participating in an advanced course sequence in high school? 

The direct effects of this intervention on these outcomes for AR students are of primary concern 
for determining whether this intervention is an effective way for schools to broaden access to 
Algebra I. 

To test the primary research questions, the study team measured algebra achievement with a 
computer-adaptive assessment. It defined advanced coursetaking as completion of a grade 9 
course above Algebra I; success in the grade 9 course above Algebra I (a grade of C or higher); 
and enrollment in the next course in the sequence in grade 10. This definition was based on 
research on mathematics coursetaking using national samples (Stevenson et al. 1994; Schneider 
et al. 1998) that defined advanced coursetaking as the completion of Algebra II by the end of 
grade 10. Because this study followed students only until the end of grade 9, grade 9 actual 
coursetaking and grade 10 planned coursetaking were used as the indicators of the likelihood of 
participating in an advanced sequence through high school. 17 

Secondary Research Questions 

Four secondary research questions are designed to test whether there are significant side effects 
of adopting an online Algebra I course for the purpose of broadening access for AR students: 

3. What is the effect of providing online Algebra I to AR students on their general 
mathematics achievement at the end of grade 8? 

4. What is the effect of providing online Algebra I to AR students on the algebra 
achievement of N-AR students at the end of grade 8? 

5. What is the effect of providing online Algebra I to AR students on the general 
mathematics achievement of N-AR students at the end of grade 8? 

6. How does offering an online Algebra I course to AR students affect the likelihood 

1 8 

that N-AR students participate in an intermediate course sequence in high school? 

To test the secondary research questions, the study team measured general mathematics and 
algebra achievement with a computer-adaptive assessment administered to both AR and N-AR 
students. It defined intermediate coursetaking as the projected completion of Algebra I in grade 
9. 


17 The advanced sequence reflects a traditional coursetaking sequence that goes from Algebra I to Geometry to 
Algebra II to Trigonometry/Precalculus to Calculus, in which each course is a prerequisite for the next. This is the 
sequence researchers expected to find in most high schools in Maine and Vermont. The definition was applied 
based on the actual course sequences in the high schools AR students attended in grade 9, as explained in chapter 2. 

18 Although researchers did not follow N-AR students’ coursetaking in grade 9, they did determine at the end of 
grade 8 whether they appeared likely to follow an intermediate high school course sequence the following fall. 
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Exploratory Research Questions 

The study also posed two exploratory questions that further examine the impact of offering 
online Algebra I to AR students on their high school coursetaking patterns and progression: 

7. How does access to online Algebra I in grade 8 affect the likelihood that AR students 
sign up for advanced courses in grade 9? 

8. How does access to online Algebra I in grade 8 affect the likelihood that AR students 
“double up,” or take more than one mathematics course per year, in grade 9 or 10? 

Overview of the Study's Evaluation Design 

This study is a randomized controlled trial with randomization at the school level. Schools in 
Maine and Vermont that did not offer a full section of Algebra I to grade 8 students (as of the 
2007/08 school year) were eligible for the study. Sixty-eight eligible schools were randomly 
assigned to one of two study groups. Schools in the treatment condition received the online 
algebra course for the 2008/09 school year; schools in the control condition did not receive the 
online Algebra I course during the 2008/09 school year. 19 To estimate the impacts of online 
Algebra I on relevant outcomes for both AR and N-AR students, researchers compared 
outcomes for students in treatment schools and students in control schools at the end of grade 8 
(spring 2009) and (for AR students only) at the end of grade 9 (spring 2010). The observed 
impacts will help determine whether adopting an online Algebra I course is a good choice for 
schools that do not offer Algebra I to grade 8 students. 

For the AR sample, this study was thus also longitudinal in design. The premise of “pushing 
down” Algebra I to grade 8 is that it prepares students for more rigorous coursetaking in 
mathematics through high school, which better prepares them to succeed in advanced course 
sequences that prepare them for college. Therefore, of critical interest from a policy perspective 
is the extent to which offering an online algebra course to students who would otherwise not 
have been able to take Algebra I in grade 8 has a sustained impact on their mathematics 
coursetaking in high school. For this reason, researchers tracked the AR students who attended 
participating schools into high school to collect mathematics coursetaking information (grade 9 
courses and grades and grade 10 planned courses) at the end of grade 9. These data were used to 
categorize students as participating in an advanced mathematics course sequence in high school. 

Researchers did not follow N-AR students into high school because of cost constraints (there 
were approximately three times as many N-AR students as AR students) and because assessing 
the impact of the online Algebra I course on AR students’ subsequent high school coursetaking 
was most critical and relevant for the study. 

Evaluation Framework 

This study compared AR students in treatment schools (who are offered the online Algebra I 
course) with AR students in control schools (who are not offered the online Algebra I course and 
presumably took a general mathematics course). The study did not make a direct contrast 
between online Algebra I and standard, face-to-face delivery of Algebra I. The comparison of 


19 Schools in the control group received the online course for the 2009/10 school year. All schools (treatment and 
control) were provided the online course for two consecutive years. 
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AR students in treatment schools to AR students in control schools was designed to reveal the 
effects on students’ mathematics achievement, over time, of using an online course in order to 
broaden access to Algebra I for students who otherwise would not have access to a formal 
Algebra I course. The study also compared N-AR students in treatment schools with N-AR 
students in control schools (figure 1-1). All N-AR students were expected to take the regular 
general mathematics course. 

Figure 1-1. Framework for Estimating Impacts of Online Algebra I on AR and N-AR 
Students 
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Definition of a Successful Intervention 

The online Algebra I course was considered successful if the results showed both 

• A statistically significant positive impact on either of the two primary research questions 
(AR students’ algebra scores at the end of grade 8 or high school coursetaking). 

• The absence of statistically significant negative side effects (as assessed by the four 
secondary research questions). 20 

Only this combination of results provides evidence that there are benefits of adopting an online 
Algebra I course for AR students without significant negative consequences to them or their N- 
AR peers. 

Content and Organization of This Report 

The findings in this report focus on the impacts of online Algebra I at the end of the school year 
during which the intervention was implemented (2008/09) and on coursetaking in the year 
following implementation. Chapter 2 describes the research design and methodology; chapter 3 
describes the implementation of the intervention; chapter 4 analyzes the impacts of the 
intervention on short- and longer-term outcomes; chapter 5 presents exploratory analyses related 
to AR students’ coursetaking; and chapter 6 summarizes key findings. Appendixes provide 


20 For the secondary questions, the study was not designed to determine whether the groups are statistically 
equivalent. A lack of statistical significance for an impact estimate does not mean that the impact being estimated 
equals zero. Rather, it means that the estimate cannot reliably be distinguished from zero, an outcome that may 
reflect the small magnitude of the impact estimate, the limited statistical power of the study, or both. 
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additional detail on the study samples, the data collected and measures used, the intervention, 
and the statistical estimation methods used. 
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Chapter 2 Study Design and Methodology 

This chapter summarizes how schools were recruited and randomly assigned to condition. It 
describes the characteristics of the participating schools, teachers, and students at the beginning 
of the study. The chapter also provides an overview of the data collected from participants during 
the study, provides baseline comparisons of the treatment and control groups (at random 
assignment and at analysis), and describes the analysis of these data to address the study’s 
research questions. 

Recruitment, Random Assignment, and Study Samples 

This section describes the recruitment of schools and the random assignment procedure used to 
allocate schools to treatment and control groups. It also describes the sample of schools and 
students assigned to each condition. 

Recruitment of Schools 

The target population included schools in Maine and Vennont that serve students in grade 8 and 
did not offer a stand-alone Algebra I class in 2007/08. Schools that delivered Algebra I content 
to some students by providing accelerated material in the context of the regular grade 8 
mathematics curriculum were considered eligible for the study. These two criteria — serving 
grade 8 students and not offering stand-alone Algebra I — plus the willingness to comply with the 
requirements of the study, were the only eligibility criteria for participation. 

The minimum detectable effect is the smallest true effect that has an 80% chance of being found 
to be statistically significant at the 5% level of statistical significance for a two-tailed test. When 
a minimum detectable effect is expressed in standard deviation units, it is referred to as a 
minimum detectable effect size. A minimum detectable effect size of 0.25 is considered 
necessary for the effect of an intervention to have “educational significance” (Bloom, Hill, 

Black, and Lipsey 2008). Power analyses indicated that a minimum of 60 schools were needed to 
achieve this minimum detectable effect size. (See appendix A for more information on the 
study’s power analyses.) 

The study was conducted in two states in the Northeast region, Maine and Vermont. Maine was 
chosen because in addition to relatively low overall enrollments in Algebra I among grade 8 
students (20%-25% in 2007), the state has a strong technology initiative that can support the 
infrastructure needed to offer an online course in schools. The Maine Learning Technology 
Initiative provides all grade 8 students and teachers in Maine with their own laptop computer for 
use throughout the school year, both in and out of school. Students are thus familiar with using 


21 A stand-alone class was defined as one full section of Algebra I taken by at least 20-25% of grade 8 students in 
the school, with a dedicated teacher. This proportion was derived from the percentage of grade 8 students in the 
REL-NEI region who took Algebra I as of 2007, which was 25% (U.S. Department of Education 2007a). 

22 Based on initial power analyses, the original target number of schools to participate in the study was 40. Elowever, 
initial recruitment efforts in Maine revealed that the number of grade 8 students within eligible schools was lower on 
average than the initial power estimates assumed (that is, the schools eligible for the study were smaller than 
expected). Therefore, the power calculations were revisited and the minimum number of schools was increased to 


60 . 
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computers as part of their daily educational activities (Berry and Wintle 2009; Silvemail and 
Gritter 2007). The technology infrastructure in Maine helps drive interest in online courses. 

To identify eligible schools in Maine, researchers used the Common Core of Data to find schools 
that served grade 8 students during the 2007/08 school year. Initial recruitment — including 
determination of eligibility and interest in participation — focused on all public schools in Maine 
that serve grade 8 students (a total of 224 schools as of fall 2007). Initial recruitment efforts, 
which began in winter 2008, focused on identifying schools eligible for the study and interested 
in participating. 

Later recruitment focused on approximately 40 schools in Vermont that state administrators and 
local educators thought would most likely be eligible. Recruitment was expanded to Vermont in 
spring 2008 because the larger sample size requirements could not be met in Maine alone. 
Vennont was selected because it shares demographic and geographic characteristics with Maine 
that were factors in the selection of Maine for the study. Specifically, Vermont has the second- 
highest proportion of rural schools in the United States, after Maine (Johnson and Strange 2007) 
and these schools serve students who, although racially and ethnically homogenous, are diverse 
in socioeconomic status. Like rural schools in Maine, rural schools in Vermont find it 
challenging to offer a full range of courses to students who might benefit from them. Although at 
the time of the study Vennont did not have the laptop initiative that Maine had, the state had 
both the necessary technological capacity and an interest in exploring ways to use technology to 
improve education. Initial contacts with state-level administrators and staff of the state education 
agency indicated a strong interest in the study, and REL-NEI has strong connections with 
policymakers, administrators, and mathematics educators in the state. 

School recruitment for this study involved a planned sequence of outreach at the state, district, 
and school levels. The partnerships REL-NEI had already established in Maine and Vermont 
facilitated these contacts. Initial recruitment outreach began at the state level. It involved 
obtaining endorsements of the study from the Commissioner of Education in Maine and the 
Acting Commissioner in Vermont. Study leaders attended meetings with state- and district-level 
staff in both states to achieve buy-in for the study and gather information regarding which 
schools to target. 

In April 2008 (the spring before implementation), the study team contacted — by phone and in 
person — the 224 schools in Maine that served grade 8 students to determine eligibility and assess 
interest in participation. Meetings were held with school principals and often included grade 8 
mathematics teachers. In May 2008, when recruitment was expanded to Vermont, officials in the 
state department of education and local educators helped the study team identify schools they 
believed were most likely to meet the two eligibility requirements for the study (that is, serving 
grade 8 students and not offering a separate Algebra I course). An initial pool of approximately 
40 Vermont schools was selected, and recruitment activities focused on these schools. 

Protocols that included talking points, scripts, and a checklist of items to be addressed guided all 
communications with schools. Class.com, the online Algebra I course provider, provided web- 
based course demonstrations that allowed principals, mathematics educators, school 
technologists, parents, and community members to see the course and ask questions about it or 
the research study. Class.com staff members were also available for briefings and question and 
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answer sessions with state- and district-level mathematics supervisors and policymakers, 
including the REL-NEI governing board. 

During recruitment, the study team informed schools of the requirements they would have to 
meet if they were randomized to the treatment group. First, students taking the online Algebra I 
course would take the course as their grade 8 mathematics course, not as a supplemental course 
to the general grade 8 mathematics class. Second, each participating student would have access 
at school to a computer with a high-speed Internet connection. Access to a computer was 
necessary for students to access the course and communicate with the online teacher. 23 Third, 
schools were required to assign a class period during the school day during which students would 
access the online Algebra I. The online class had to occur with the same frequency and duration 
as the regular grade 8 mathematics classes, but schools could schedule the period whenever they 
wanted, create multiple periods during the school day, and determine the location of the online 
course (for example, the regular classroom or the library). Fourth, schools would provide a staff 
member to serve as an on-site proctor to support the students taking the online course. 

Schools signed memoranda of understanding to signal their agreement with the requirements of 
participation in the study. The tenns and conditions of the memorandum of understanding 
included clear infonnation on random assignment. Moreover, all schools were required to 
provide a list of AR students who would be eligible for the intervention should the school be 
assigned to receive the course during the 2008/09 school year. This requirement ensured that 
schools’ identification of AR students was unaffected by their assignment to the treatment or 
control group. 

By mid- June 2008, 71 schools in Maine and Vermont signed memoranda of understanding 
indicating their interest in participating in the study. Among the schools that agreed to 
participate, three schools withdrew from the study before or after random assignment. Reasons 
for withdrawal included lack of identification of AR students before random assignment and 
principal turnover. Following these exclusions, 68 eligible schools were randomly assigned to a 
condition and participated in the study. 

Description of Participating Schools 

The 68 schools that were randomly assigned are spread across Maine and Vermont. Sixty-two 
are rural schools, as defined by the Common Core of Data. As of the 2007/08 school year, grade 
8 enrollments ranged from fewer than 4 to nearly 150 students, with an average enrollment of 32 
students. Fifty-two schools (76%) served grades pre-K-8 or K-8; 10 schools (14%) served 
middle grades (grades 5-8, 6-8, or 7-8); and the remaining six schools serve other grade spans 
including K-12, 7-12, and 3-8. 

To take the online course, students were expected to have been deemed “algebra ready” before 
random assignment (that is, as of the end of grade 7). AR students were students whom teachers, 
principals, and guidance counselors perceived as having the requisite skills at the end of grade 7 
to take Algebra I in grade 8. Schools made decisions about which students were ready for 
algebra on the basis of teacher perceptions of preparedness; grades in mathematics classes 


23 Technology requirements and system specifications were provided to all schools during recruitment and again to 
the schools in the treatment group before the beginning of the 2008-09 school year. 
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through grade 7; scores on state assessments; and scores on other assessments, such as algebra 
readiness tests. The research team did not impose a definition or set of criteria for “algebra 
readiness” on the participating schools for two main reasons. First, there were no common 
instruments across all schools that were administered prior to random assignment that were 
specifically measures of algebra readiness. Second, the study aimed to test the effectiveness of 
offering an online Algebra I course in a real-world context, where local decisionmaking about 
student eligibility for the course would be the norm. 24 

Before random assignment, the 68 schools allocated to treatment or control groups identified 468 
AR students, an average of 6.9 students per school. These students represented 23% of the rising 
grade 8 students in participating schools as of spring 2008. 

Random Assignment Procedure 

The schools were stratified by two blocking variables: state and school size. The blocking 
variables were selected with the expectation that the variability across schools would be smaller 
within state and size blocks than in the full sample. The state blocks had two categories: Maine 
and Vermont. The size block had three levels, based on the size of the entering grade 8 student 
population: small (fewer than 17 grade 8 students), medium (17-70 grade 8 students), and large 
(more than 70 grade 8 students). Half of the schools within each state by size block were 

25 

randomly assigned to the treatment group (table 2-1). 


Table 2-1. Number of Treatment and Control Schools, by Enrollment Size 


Block 

State 

Size 

(based on grade 8 student 
population) 

Number of 
treatment 
schools 

Number of 
control 
schools 

Total 

number of 
schools 

1 

Maine 

Small (fewer than 17) 

12 

12 

24 

2 and 3 

Maine 

Medium (17-70) and large 
(71 or more) 

13 

13 

26 

4 

Vermont 

Small (fewer than 17) 

5 

3 

8 

5 and 6 

Vermont 

Medium (17-70) and large 
(more than 70) 

5 

5 

10 

Total 



35 

33 

68 


Note: Sample included 68 schools (35 treatment, 33 control). To maintain the confidentiality of participating schools, 
researchers aggregated blocks that included fewer than three schools for presentation purposes only. 

Source: Study records. 


24 Because schools used their own criteria for identifying AR students, it is possible that the AR and N-AR student 
samples would not have been distinct in terms of prior achievement, as intended. The differences at baseline 
between the AR and N-AR samples are presented in a section below on Sample Characteristics at Random 
Assignment. (Details on differences between the AR and N-AR samples are also provided in appendix A.) 

25 When the number of schools within a block was odd, a coin was flipped to determine whether the last school 
would be assigned to the treatment or control group. 
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Participating schools were infonned of their treatment or control status in late June 2008. 

Schools that were randomized to the treatment group were informed that their AR students 
would have the option to take the online Algebra I course during the 2008/09 school year. 
Schools randomized to the control group were informed that their school would conduct business 
as usual during the 2008/09 school year. All 68 schools cooperated with data collection and fully 
participated in the study during the 2008/09 school year. This participation included 
implementation of the online Algebra I intervention in all 35 schools that were randomized to the 
treatment condition. 

Student Samples 

The study included two samples of grade 8 students: AR students and N-AR students (table 2-2). 
At the start of the 2008/09 school year, 445 AR students were enrolled in the 68 schools 
randomly assigned to either treatment or control, because 23 AR students had moved over the 
summer (12 from treatment schools and 1 1 from control schools). The average number of AR 
students per school was 6.5. AR students represented 22% of grade 8 students in participating 
schools. In four small schools, all grade 8 students were identified as AR students. 

N-AR students were identified by collecting rosters of grade 8 mathematics classes from all 
study schools at the start of the 2008/09 school year. The sample included 1,554 students 
enrolled in study schools at that time. Students who arrived in the study schools after the start of 
the school year (that is, after pretesting was complete) were excluded from the analysis. 


Table 2-2. Number of Schools and Students per Condition as of Fall 2008 


Item 

Total 

Treatment 

Control 

p-value 

Number of schools 

68 

35 

33 

a 

Number of grade 8 AR students 

445 

218 

227 

0.670 

Number of grade 8 N-AR students 

1,554 

782 

772 

0.800 

Total number of grade 8 students 

1,999 

1,000 

999 

0.982 

Average number of AR students per 
school (standard deviation) 

6.54 (5.23) 

6.23 (5.21) 

6.73 (532) 

0.698 

Average number of grade 8 students per 
school (standard deviation) 

31.94 (37.01) 

31.00 (40.93) 

32.94 (32.96) 

0.830 


AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,999 students (445 AR students, 1,554 N-AR 
students). AR students were identified before random assignment in June 2008. The sample of AR students does 
not include 23 students who moved during summer 2008; it does include 5 students who later refused to participate 
in the study. The number of N-AR students is based on information obtained from school rosters in fall 2008. The 
N-AR sample includes 63 students who later refused to participate in the study and 46 students who were later 
deemed “not testable” by their teachers and schools. Tests of significance were conducted using two-tailed y2 and 
independent sample t-tests. 

a. Not applicable, because schools were allocated to treatment and control using a block randomized procedure. 

Source: Records obtained from each school before random assignment (June 2008) and school rosters examined in 
fall 2008. 

Enrollment in the Online Course 

More than 96% of the AR students in treatment schools (211 of 2 1 8) enrolled in the online 
Algebra I course for the 2008/09 school year (table 2-3). In control schools, none of the 227 AR 
students enrolled in the online Algebra I course, which was not offered at their schools. There 
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were no student crossovers (AR students from control schools who moved to a treatment school 
and enrolled in the course). 

Of the 782 N-AR students in treatment schools, 3 1 enrolled in the Algebra I online course." The 
remaining N-AR students in treatment schools, and all of the 772 N-AR students in control 
schools, did not enroll in the course. 


Table 2-3. Allocation of Students in Treatment and Control Schools, as of Fall 2008 


Type of student 

Number of students in 
treatment schools 

Number of students in 
control schools 

AR 

218 

227 

Enrolled in the online course 

211 

0 

Not enrolled in the online course 

7 

227 

N-AR 

782 

772 

Enrolled in the online course 

31 

0 

Not enrolled in the online course 

751 

772 

Total grade 8 students 

1,000 

999 


AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,999 students (445 AR students, 1,554 N-AR 
students). The sample of AR students does not include 23 students who moved during summer 2008; it does 
include 5 students who later refused to participate in the study. The N-AR sample includes 63 students who 
later refused to participate in the study and 46 students who were later deemed “not testable” by their teachers 
and schools. Students who moved into study schools during the 2008/09 school year were deemed ineligible 
because they were not classified as algebra ready or not algebra ready before random assignment. 

Source: Records obtained from each school before random assignment (June 2008) and school rosters 
examined in fall 2008. 

Consent 

At the beginning of the 2008/09 school year, information on the study was sent home to parents 
of all grade 8 students in the schools, with a form to complete if parents wanted to withdraw their 
children from participating in data collection activities. Sixty-eight students were withdrawn 
from data collection (29 N-AR students in treatment schools and 5 AR students and 34 N-AR 
students in control schools; see table 2-4). 


Table 2-4. Students Not Granted Parental Permission to Participate in the Study 



Treatment 

Control 


Type of student 

Number 

Percent 

Number 

Percent 

/j-value 

AR (n = 445) 

0 

0 

5 

2.2 

0.144 

N-AR (n - 1,554) 

29 

3.7 

34 

4.4 

0.455 


AR is algebra ready. N-AR is not algebra ready. 

Note: Tests of significance were conducted using hierarchical linear models that adjusted for the nesting of students within 
schools. 

Source: Study records. 


26 Schools were allowed to enroll additional students in the online course. The 3 1 N-AR students who enrolled in 
the online course did so because their teachers, their parents, or the students themselves believed that the course was 
appropriate for them, even though they were not initially considered algebra ready when schools were asked to 
submit their lists. Although these students participated in the intervention, they are not part of the AR student sample 
and remain in the non-algebra ready student sample for analysis. 
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In preparation for administration of the pretest, a second set of students was withdrawn from data 
collection. These students were deemed “not testable” by their schools and teachers, because 
given these students’ special education status, appropriate accommodations could not be 
provided for the pretest and posttest administered as part of the study. All “not testable” students 
were in the N-AR student sample (9 in treatment schools, 37 in control schools). Like the 
students whose parents withdrew them from data collection, they are considered out of scope of 
the analytic sample. 

For the 2009/10 follow-up data collection of high school course information for the AR student 
sample, 10 of 66 high schools required additional consent from parents to release student data. In 
these schools, one of the following consent procedures was used: the study team worked with the 
schools to obtain written documentation of consent, the study team sent additional infonnation 
home to parents with a form to complete if parents wanted to opt out, or the principal contacted 
parents informally to ask for permission to participate. 

Figure 2-1 presents the study sample through the study, from recruitment to analysis. (See 
appendix A for detailed information on the sample). 
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Figure 2-1. Consolidated Standards of Reporting Trials (CONSORT) Diagram 


Excluded 

School n- 3 

(dropped out or ruled ineligible) 


(spring 2008) 

School n =35 
AR student n = 230 


Recruited 

School n- 71 



Randomized and Eligible 
Schools and Students 


T 

School n = 68 
AR student n = 468 

▼ 

1 77. : 7~. 


(spring 2008) 

School n = 33 
AR student n = 238 


Treatment Sample Tracking 
(fall 2008) 

AR student n = 230 

Moved (dropped) n = 12 
Refused to participate n = 0 
N-AR student n = 744 a 


Control Sample Tracking 
(fall 2008) 

AR student n = 238 

Moved (dropped) n = 11 
Refused to participate n = 5 
N-AR student n = 701 a 


Treatment Baseline Data 

School n =35 
AR student n = 217 
N-AR student n = 714-721 


Control Baseline Data 

School n =33 
AR student n = 218-219 
N-AR student n = 670-682 


Treatment Outcome Data 

School n =35 
AR student n = 211-216 
N-AR student n = 630-697 


Control Outcome Data 

School n =33 
AR student n = 211-218 
N-AR student n = 611-652 


Treatment Analyzed 

School n =35 

AR student analysis sample n = 218 
N-AR student analysis sample n = 744 


Control Analyzed 

School n =33 

AR student analysis sample n = 222 
N-AR student analysis sample n = 701 


a. The N-AR sample was 1,554 students. However, due to the withdrawal of students from data collection by 
parents ( n = 63) and schools ( n = 46), the total N-AR student sample allocated to treatment and control was 1,445 
with 744 students allocated to treatment and 701 students allocated to control. 

b. The reported ranges for the baseline sample sizes are the ranges of nonmissing data for the two baseline 
measures, state mathematics assessments and the study-administered general mathematics pretest. Study- 
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administered pretest scores that were based on testing time of less than five minutes were excluded from the 
sample size ranges. 

c. The sample size ranges reported for the outcome data are the ranges of nonmissing data for the three outcome 
measures (the end of grade 8 algebra posttest, the planned grade 9 courses, and high school coursetaking 
information [collected for AR students only]). Posttest scores that were based on testing time of less than five 
minutes were excluded from the sample size ranges. 

d. Multiple imputation was used for data analysis. The number of imputed cases ranged from fewer than 4 to 7 for 
the AR treatment sample, depending on the outcome, and from 47 to 114 for the N-AR treatment sample, 
depending on the outcome. The number of imputed cases was 4-1 1 for the AR control sample, depending on the 
outcome, and 49-90 for the N-AR control sample, depending on the outcome. 

Source : Study records. 

Data Collected for the Study 

The study data serve four main purposes. They document the implementation of the online 
Algebra I course and the general mathematics course in comparison schools, provide descriptive 
in formation on sample characteristics, provide covariates for the outcome analyses, and serve as 
the study outcomes. This section briefly overviews the data collected (appendix B provides more 
detailed information on the measures used). Table 2-5 lists the main sources of data for the study 
and the timeline for data collection. 
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Table 2-5. Data Sources and Collection Schedule 


Type of data 

Primary purpose 

Data collection schedule 

Preintervention 

Intervention 

Postintervention 

Sample 

characteristics/ 

covariates 

Document 

implementation 

Measure 

outcomes 

Spring 2008 

Fall 

2008 

Winter 

2009 

Spring 

2009 

Fall 

2009 

Winter 

2010 

Spring 

2010 

School characteristics 

✓ 



✓ 







Student characteristics 

S 



✓ 







State assessment scores 
(prior mathematics 
achievement) 

S 










Study-administered pretest 
(general mathematics) 

V 




y 






Site visits 


✓ 



s 

S 

✓ 




Online course activity data 


✓ 



s 

•/ 

V 




Proctor logs 


✓ 



s 


V 




Teacher survey 

V 






V 




Classroom materials 


✓ 





S 




Study-administered 
posttests (Algebra I and 
general mathematics) 



■/ 




S 




Planned grade 9 
mathematics courses 







■/ 




High school characteristics 
and course offerings 








S 


•/ 

Grade 9 course types and 
grades (AR students only) 



■/ 







y 

Planned grade 10 
mathematics courses (AR 
only students) 



■/ 







V 


AR is algebra ready. 




Measures of Sample Characteristics and Covariates 

• School characteristics 

o Middle schools. Researchers obtained demographic characteristics of the participating 
schools — including total enrollment, the demographic composition of the student 
body, the percentage of students eligible for free or reduced-price lunch. Title I status, 
and grade span for the year before the study — from the Common Core of Data. These 
data were used in a baseline (preintervention) comparison of schools in the treatment 
and control groups. 

o High schools. Data were also collected on the 66 high schools that AR students 
attended in 2009/10 as grade 9 students. Forty-five of these high schools were public 
and 2 1 were private. Data on public high schools were collected through the Common 
Core of Data for 2007/08. Data on private high schools were collected through the 
Private School Universe Survey for 2007/08. 

• Student characteristics. Administrative records for all grade 8 students in the study 
were obtained from the Maine State Department of Education for students in Maine and 
from each supervisory union (district) for students in Vermont. Data collected included 
race and ethnicity, gender, eligibility for free or reduced-price lunch, and eligibility for 
special education services. Student-level demographic characteristics were used to 
establish baseline equivalence between students in treatment and control schools. They 
were also used as covariates in the impact analyses. Student records were available for 
more than 97% of the students attending study schools. 

• State assessment scores. Data on prior achievement on state mathematics assessments 
were collected from the Maine State Department of Education and from each supervisory 
union in Vennont. Scores were collected for all students on the state mathematics 
assessment taken the year before the study, when students were in grade 7. The Maine 
state assessment is given in the spring of each year; the scores collected were from spring 
2008. The Vermont state assessment is given in October each year; the scores collected 
were from fall 2007. Because Maine and Vennont use different tests, it was necessary to 
translate scores into a common metric. All scores were standardized to Z-scores using the 
mean and the standard deviation of the test scores within each state, including only 

97 

schools participating in the study (AR and N-AR samples combined)." 


27 Standardizing (using Z-scores) across grades and states is the most powerful and efficient way to combine results 
from different assessments (May et al., 2009). It is appropriate only if the tests are similar, the tests assess similar 
constructs across states and grades, and the samples across states and grades are similar and represent similar cross- 
sections of the population of students targeted by the intervention (May et ah, 2009). The first assumption is 
reasonable, because the Maine and Vermont state standards in middle grades mathematics are similar (Hupp, 2009) 
and the state achievement tests assess these standards. The second assumption is also reasonable, because the 
student samples in Maine and Vermont represent similar subpopulations of students demographically (the proportion 
of girls was 49% in Maine and 50% in Vermont; the proportion of ethnic/racial minority students was 4% in Maine 
and 4% in Vermont; the proportion of students eligible for free or reduced-price lunch was 41% in Maine and 46% 
in Vermont; and the proportion of students eligible for special education was 12% in Maine and 15% in Vermont), 
and in terms of the underlying construct of mathematics achievement, as measured by the Promise Assessment, the 
pretest measure administered to all study students (Maine: mean = 324.26, standard deviation = 31.20; Vermont; 
mean = 320.14, standard deviation =30.46; /? = 4.35 ,p = 0.118). (The difference on pretest was examined using a 
model that accounts for the clustering of students within schools and included both AR and N-AR students.) 
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• Promise Assessment pretest. The Promise Assessment (Internet Testing Systems and 
SEG Assessment 2009) is a computer-adaptive mathematics assessment with both a 
general mathematics (pre-algebra) item bank and an algebra item bank. The general 
mathematics item bank was used for the pretest. It was administered to all grade 8 
students in participating schools in September and October 2008. The pretest presented 
30 items that ranged in difficulty from the grade 5 to the grade 8 level. The 30 items were 
targeted to ability level and drawn from a test bank of approximately 1,000 items. The 
items were distributed across six domains (number, computation and estimation, 
measurement, geometry, probability and statistics, and algebraic concepts). The 
reliability for the 30-item general mathematics Promise Assessment in the calibration 
group is 0.87. Scores for the Promise Assessment are generated by a linear 
transformation of the underlying Rasch scale and are reported on a scale of 200 to 400. 
Response rates on the pretest were 99% for AR students and 97% for N-AR students. 
(See appendix B for more information on the Promise Assessment.) 

Measures of Implementation 

• Online course activity data. The online course management system that Class.com used 
to deliver the course (Moodle) automatically logs and stores data documenting online 
course activity of students, online teachers, and proctors. For all online course sections, 
Moodle records the date and time that users logged into the system, as well as the sender, 
recipient, and content of all messages. The software also records the content accessed by 
students, as well as quiz and exam grades. (Chapter 3 provides details on the information 
recorded by the course system.) The system also automatically logs records of teachers’ 
and proctors’ review of student records. These logs were the primary source of 
information used to assess online teachers’ monitoring of and communication with 
students. The study team “observed” direct online interactions between online teachers 
and their students using these archived data. For each of the 10 online course sections, the 
study team randomly selected one school day each month (October-May) and 
downloaded all online activity over a 24-hour period. A 24-hour period was observed 
because it was the amount of time within which online teachers were expected to respond 
to communications from individual students. Researchers coded the archived data to 
indicate whether the online teachers logged into the course during each 24-hour period 
and whether they posted announcements on the home page for their sections; they also 
recorded the number and type of communications between teachers and students. They 
collected the 24-hour sample of archived data once a month for 8 months for each of the 
10 course sections, for a total of 80 observations. Archived online course data were also 
the primary source of information on students’ progression through the course (that is, 
course completion). 

• Site visits. Study team members visited every treatment school once during the 2008/09 
school year, in order to assess the extent to which the online course was being 
implemented as intended. At each school, researchers noted the physical location of 
students taking the online course and proctor activities while students accessed the 
course. (A copy of the protocol used for these visits is provided in appendix B.) 

• Proctor logs. Proctors who supervised the students in the online course were asked to 
complete a web-based log once a week to record the amount of time they spent 
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performing specific types of activities as part of their proctor role. Data from the logs 
were used to monitor implementation across treatment schools. All proctors completed at 
least half of their weekly logs. (A description of the proctor logs is provided in appendix 
B.) 

• Teacher survey. Researchers administered a web-based teacher survey in spring 2009 to 
all grade 8 mathematics teachers in study schools and to online Algebra I teachers. The 
survey served two main purposes. First, it provided data on characteristics of the teachers 
(for example, degree earned, years of teaching experience). Second, it provided data on 
the delivery of grade 8 mathematics instruction in treatment and control schools, as well 
as in the online Algebra I classes. The response rate on the teacher survey was 95%. (A 
description of the teacher survey and the key constructs used in this report is provided in 
appendix B.) 

• Classroom materials. To describe the characteristics of the regular grade 8 mathematics 
classes in study schools, the study team used a structured protocol to collect instructional 
materials, including teacher-provided information on the textbook used and actual course 
syllabi, exams, and pacing guides (if available). The study team coded these materials for 
content by using the list of mathematics concepts included on the teacher survey. It then 
calculated the proportion of concepts covered to estimate the amount and type of 
algebraic concepts taught in the general grade 8 mathematics classes. Classroom 
instructional materials were obtained for 90% of grade 8 mathematics classes in study 
schools. (Appendix B describes the collection and coding of classroom materials.) 

Outcome Measures 

Promise Assessment posttests. The Promise Assessment posttest was administered to all grade 
8 students in participating schools in May and June 2009. The posttest was delivered as a 40- 
item that included 20 items from the general mathematics item bank and 20 items from the 
algebra item bank. 

The algebra item ha nk contains approximately 300 items. The reliability of the 20-item test 
drawn from this bank in the calibration group is 0.80. Scores on the algebra posttest are reported 
on a scale of 400 to 500. These scores represent the primary achievement measure at the end of 
grade 8. Scores on the general mathematics posttest, reported on a scale of 200 to 400, represent 
a secondary measure of achievement at the end of grade 8. The reliability for the 20-item general 
mathematics assessment in the calibration group is 0.83. Response rates on the algebra and 
general mathematics posttests were 99% for AR students and 94% for N-AR students. However, 
scores based on less than five minutes of testing were determined to be invalid by the test 
developer and thus were dropped and treated as missing. For the algebra posttest, there were 
fewer than four such cases in the AR sample and 1 18 in the N-AR sample (73 in treatment and 
45 in control). Eight percent of the algebra posttest scores were cut from the N-AR sample 
because of testing time of less than five minutes. 

High school mathematics coursetaking. To measure coursetaking after grade 8, the study team 
drew on two sources of data. For both AR and N-AR students, it gathered data on students’ 
planned grade 9 mathematics classes. For AR students only, it gathered actual course enrollment 
and grades for grade 9 and planned coursetaking for grade 10. 
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• Planned grade 9 mathematics classes (collected at the end of grade 8) for all 
students. In spring 2009, study team members collected planned grade 9 course 
enrollment information for all grade 8 students in study schools. These data were 
provided by participating schools, which in many cases contacted the high schools 
students planned to attend to obtain the infonnation. The study team collected the name 
of the high school and the course title for the grade 9 mathematics course in which each 
student was enrolled for fall 2009. These data were made available to the study team for 
97% of AR and 93% of N-AR students. 

• Grade 9 and 10 mathematics course sequences (collected at the end of grade 9) for 
AR students. In spring 2010, for all AR students in the sample, the study team collected 
course titles and grades for the mathematics courses taken in grade 9 and course titles for 
the mathematics courses in which students planned to enroll for grade 10. These data 
were collected from the high schools AR students attended in grade 9. Codeable data for 
427 AR students (97% of the sample) were obtained. These data were not collected for 
the N-AR students because of cost constraints and the determination that assessing the 
impact of online Algebra I in grade 8 on subsequent coursetaking was most critical and 
relevant for the AR students. 

Coding of high school coursetaking data was based on methods used by the National Center for 
Education Statistics for the National Assessment of Educational Progress and Education 
Longitudinal Study transcript studies. Transcript coding protocols guided the extraction of 
course identifiers. Mathematics education experts coded the course titles, using the Classification 
of Secondary School Courses, which is based on information available in school catalogs and 
other information sources (U.S. Department of Education 2007b). 

The creation of the coursetaking indicators based on these codes was guided by previous 
research on typical high school course sequencing and definitions of “advanced,” “intermediate,” 
or “low” high school course sequences by Schneider and colleagues (1998) and Stevenson and 
colleagues (1994). In U.S. high schools, the typical sequence is Algebra I -> Geometry -> 
Algebra II -> Pre-calculus/Trigonometry -A Calculus. Advanced, intermediate, and low 
sequences are defined by where students are in this pipeline during each year of high school. The 
study team drew on this research to define two coursetaking sequences for the study: “advanced” 
for AR students and “intermediate” for N-AR students. 

For AR students (who were followed into high school), the study team coded whether their 
actual grade 9 courses and grades and planned grade 10 courses were indicative of an advanced 
sequence, defined by Schneider and colleagues (1998) as the successful completion of Geometry 
and Algebra II by grade 10. 
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AR students who met the following criteria were coded as participating in an advanced course 
sequence: 

• Completed a full-year course above Algebra I or equivalent in grade 9. 

• Earned an end-of-year grade of C or above in the grade 9 course (if more than one grade 
9 course was taken, the grade had to be C or higher in the most advanced course taken to 
meet this criterion). 

• Enrolled in Algebra II (or the next course in the sequence) for grade 10. 

Students who did not meet all three criteria were coded as not participating in an advanced 
course sequence. 28 

The collection and coding of coursetaking for N-AR students was different from that for AR 
students in two respects. First, the study team collected data about planned courses for N-AR 
students at the end of grade 8 but did not follow these students into high school. Second, because 
these students were not identified as ready for algebra as rising grade 8 students, researchers 
assumed that they would not follow an advanced sequence in high school. They therefore coded 
whether their planned grade 9 courses were indicative of an intennediate sequence, defined by 
Schneider and colleagues (1998) as the successful completion of Algebra I in grade 9 and 
Geometry in grade 10. N-AR students were assigned codes for planned grade 9 courses 
according to whether or not the course for grade 9 was at or above Algebra I. (Appendix B 
provides more information on the collection and coding of high school courses.) 

Characteristics of the Study Sample 

This section examines baseline equivalence at three time points: (1) at the time of random 
assignment; (2) at the time of analysis, with no imputation of missing cases; and (3) at the time 
of analysis, where missing data are imputed. 

School Characteristics at Random Assignment 

Table 2-6 summarizes the characteristics of the sample of study schools in the year before the 
study. Baseline measures, pertaining to the 2007/08 school year, indicate the following: 

• The average total enrollment for study schools was 186 students. Average grade 8 
enrollment was 32. 

• 94% of the schools were Title I schools. 

• 48% of the students were eligible for free or reduced-price lunch. 

• 95% of the students were white. 

• 53% of the students scored at or above the proficiency level on the state mathematics 
assessment in the 2007/08 school year. 


2S For AR students, the study team also collected and coded the planned grade 9 courses (based on end-of-grade 8 
enrollments) as a binary variable representing initial registration in a course above Algebra I versus at or below 
Algebra I. Chapter 5 reports the impact of online Algebra I on AR students’ initial course registrations. 
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Table 2-6. Baseline Middle School Characteristics in 2007/08 before Random Assignment 



All schools 
(« = 68) 

Treatment schools 
(« = 35) 

Control schools 
(// = 33) 


Characteristic 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

Mean 

Standard 

deviation 

/?-value 

Eligible for free or reduced-price lunch (average percent) 11 

48 

15.73 

50 

14.49 

47 

16.94 

0.337 

Race/ethnicity (average percent) 
White 

95 

11.08 

95 

5.93 

94 

14.82 

0.716 

Black 

1 

2.55 

2 

3.42 

1 

0.80 

0.122 

Hispanic 

1 

2.06 

1 

2.41 

1 

1.66 

0.42 

Asian/Pacific Islander 

1 

1.48 

1 

1.90 

1 

0.83 

0.353 

Other 

2 

10.32 

1 

1.87 

3 

14.69 

0.332 

Female (average percent) 

48 

5.08 

47 

4.56 

48 

5.63 

0.705 

Proficient on state mathematics assessment (average 
percent 

53 

13.71 

53 

11.32 

53 

16.04 

0.947 

Student-teacher ratio 

8.37 

2.46 

8.50 

2.43 

8.24 

2.52 

0.786 

Number of grade 8 students 

32 

37.00 

31 

40.93 

33 

32.96 

0.889 

Total school enrollment 

186 

133.86 

180 

130.89 

192 

138.67 

0.773 


Percent 

Standard 

deviation 

Percent 

Standard 

deviation 

Percent 

Standard 

deviation 

/;-value 

Title I status (percent of schools) 

94 

na 

t 

na 

t 

na 

0.288 


na is not applicable. 

a. Data were missing for fewer than four schools 


Note : Sample includes 68 schools (35 treatment, 33 control). Values are unadjusted. Logistic regression models with treatment status as the dependent variable were used to 
determine whether the means for the study groups were significantly different from one another, 
f To maintain confidentiality of participants, the percentage was suppressed for presentation purposes. 

Source: School-level data obtained from the Common Core of Data for school year 2007/08, 



Table 2-7 summarizes the characteristics of students in the participating schools as of fall 2008. 
It shows baseline characteristics for the AR and N-AR samples overall (across conditions) and 
by condition. 


Table 2-7. Baseline Student Characteristics of AR and N-AR Student Samples at Random 
Assignment (Missing Data Not Imputed) 


Characteristic 

Overall 

Treatment 

Control 

/i-value 

AR students 

Percent eligible for free or reduced-price lunch ( n = 436) 

32 

34 

30 

0.596 

Percent receiving special education services (n = 437) 

3 

3 

4 

0.622 a 

Percent limited English proficient ( n = 437) 

3 

5 

2 

0.646 a 

Percent female (n = 440) 

49 

49 

49 

0.833 

Percent racial/ethnic minority ( n = 440) 

7 

8 

5 

0.975 

Mean grade 7 score on state mathematics assessment 

0.95 

0.97 

0.94 

0.584 

(standardized) 13 ( n = 437) 

(0.69) 

(0.59) 

(0.77) 



349.87 

353.61 

346.14 

0.035 

Mean fall 2008 Promise Assessment pretest score 13 ( n = 435) 

(23.27) 

(22.33) 

(23.64) 


N-AR students 

Percent eligible for free or reduced-price lunch ( n = 1,403) 

46 

46 

47 

0.883 

Percent receiving special education services (n = 1,419) 

17 

19 

16 

0.255 

Percent limited English proficient (n = 1,419) 

3 

4 

2 

0.927 a 

Percent female (n = 1,439) 

50 

49 

50 

0.731 

Percent racial/ethnic minority (n = 1,438) 

5 

7 

4 

0.596 a 

Mean grade 7 score on state mathematics assessment 

-0.24 

-0.25 

-0.22 

0.609 

(standardized) 13 (n = 1,403) 

(0.86) 

(0.84) 

(0.89) 



312.60 

313.72 

311.41 

0.248 

Mean fall 2008 Promise Assessment pretest score c (n = 1,384) 

(27.23) 

(27.01) 

(27.43) 



Note: AR is algebra ready. N-AR is not algebra ready. 

Sample includes 68 schools (35 treatment, 33 control); Full samples included 440 AR students (218 treatment, 222 control); 
and 1,445 N-AR students (744 treatment, 701 control); 4 control schools had no N-AR students. Student sample sizes vary for 
each row, based on the amount of missing data for each student characteristic. 

Values are unadjusted. Differences in student characteristics by condition were tested using a model that accounts for the 
clustered data structure and blocking used for randomization. Figures in parentheses are standard deviations. 

a. The model did not converge to produce estimates when controlling for five state by size dummy blocking variables. Reported 
p - value represents a model that controls for state and two dummy indicators for medium and large schools rather than their 
interactions. 

b. State mathematics scores were standardized by using the mean and standard deviation of the test scores within each state, 
including only schools participating in the study. Data were missing for 3 AR students (1 from treatment schools, and 2 from 
control schools); data were also missing for 42 N-AR students (23 from treatment schools, and 19 from control schools) 

c. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. Data were missing for 5 AR students (1 from treatment schools, and 4 from control schools); data were also missing 
for 61 N-AR students (30 from treatment schools, and 31 from control schools). 

Source: Maine state department of education and Vermont supervisory unions; study records. 


AR versus N-AR Student Sample Characteristics at Random Assignment 

In table 2-7, the characteristics of AR and N-AR students are not directly compared. Appendix 
A, table A- 1 provides details on the AR and N-AR samples including the significance tests of 
differences in their characteristics. As previously noted, decisions about which students were AR 
were made by each school, not on the basis of an objective criterion applied across schools. This 
school-determined process appeared to result in AR and N-AR samples that were statistically 
significantly different on several dimensions: 
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• 32% of AR and 46% of N-AR students in the study were eligible for free or reduced- 
price lunch. 

• 3% of AR and 17% of N-AR students received special education services. 

• On average, AR students scored nearly 1 standard deviation above the sample mean on 
their grade 7 state mathematics test (z = 0.95). The average among N-AR students was 
0.24 standard deviations below the mean. 

• In fall 2008, the average pretest score for AR students was 349.9 (standard deviation = 
23.3); for N-AR students, the average pretest score was 3 12.6 (standard deviation = 

27.2). 

Thus AR students scored significantly higher on measures of prior mathematics achievement 
than N-AR students. They were also less likely to be eligible for free or reduced-price lunch or 
to receive special education services than N-AR students. AR and N-AR students were similar 
on other demographic characteristics, including gender, race/ethnicity, and English proficiency. 

Baseline Equivalence of AR and N-AR Treatment and Control Groups at Random Assignment 

The purpose of random assignment is to produce groups that are comparable on measured and 
unmeasured characteristics, to rule out any explanations other than chance or a treatment effect 
for postintervention differences between the groups. Differences between the treatment and 
control groups did not differ significantly from zero across measured demographic 
characteristics and prior mathematics achievement based on pre-random assignment data from 
the prior school year (table 2-6 and 2-7). 

In contrast, students in treatment and control schools in the AR sample differed significantly on 
the study-administered pretest administered after random assignment, in September-October 
2008. Specifically, AR students in treatment schools outperfonned their counterparts in the 
control group. Additional sources of baseline information indicated that observed differences 
between AR students in treatment and control schools may be related to schools’ assignment to 
the treatment condition rather than chance. There was no observed difference by condition on the 
state mathematics assessment scores from the year before the study or on any other measured 
demographic characteristics that are correlated with achievement. 

In addition to an observed difference in AR students’ scores by condition on the study- 
administered pretest, there was an observed difference by condition in the amount of time spent 
by AR students on the pretest. The computer-adaptive pretest records the amount of time 
(minutes and seconds) each test-taker spent on the test. Analysis of testing times by condition 
revealed that AR students in treatment schools spent an average of approximately five minutes 
longer on the pretest than AR students in control schools ip = 0.028; see appendix B, tables B-3 
and B-4 for results of the testing time analyses). These results suggest that the study- 
administered pretest was administered to or regarded differently by AR students in treatment and 
control schools, introducing a potential bias in favor of the treatment students that does not 
reflect an actual difference in knowledge at the start of the study. To avoid this potential bias, 
researchers used the prior year’s state mathematics assessment, which was administered before 
the launching of the study, random assignment, or implementation of the intervention, rather than 
the study-administered pretest as an unbiased measure of baseline mathematics achievement. 
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Baseline Equivalence of AR and N-AR Treatment and Control Groups at Analysis 

Researchers also examined the baseline equivalence of the treatment and control groups for the 
AR and N-AR samples used in the analysis. Analyses testing baseline equivalence are provided 
with two sets of study samples. First, table 2-8 shows student characteristics for the AR and N- 
AR student samples that include all students with complete data on the primary outcomes 
(algebra posttest scores and high school coursetaking codes). Second, table 2-9 shows student 

29 

characteristics for all students included in the main analyses, where missing data were imputed." 

Table 2-8. Baseline Characteristics of Algebra-Ready and Non- Algebra Ready Samples at 
Analysis for Students with Complete Outcome Data 


Sample with Non-Missing Sample with Non-Missing 

Algebra Scores Coursetaking Codes 

p- p- 


Characteristic 

Overall 

Treatment 

Control 

value 

Overall 

Treatment 

Control 

value 

AR students 

Percent eligible for free or reduced- 
price lunch 

Percent receives special education 

32 

34 

29 

0.484 

32 

34 

29 

0.533 

services 

3 

3 

4 

0.681 

3 

2 

4 

0.547 

Percent limited English proficient 

3 

5 

2 

0.796 

3 

4 

2 

0.706 

Percent female 

49 

49 

49 

0.836 

49 

49 

49 

0.782 

Percent racial/ethnic minority 

7 

8 

6 

0.825 

7 

8 

6 

0.774 

Mean grade 7 score on state 
mathematics assessment 

0.95 

0.97 

0.93 


0.96 

0.98 

0.94 


(standardized) 3 

(0.68) 

(0.59) 

(0.76) 

0.584 

(0.69) 

(0.59) 

(0.77) 

0.499 

Mean fall 2008 Promise 

349.74 

353.45 

346.04 


349.82 

353.52 

346.16 


Assessment pretest score b 

(23.30) 

(22.36) 

(23.69) 

0.039 

(23.53) 

(22.58) 

(23.93) 

0.047 

Total number of AR students 0 

434 

216 

218 


427 

211 

216 



N-AR students 


Percent eligible for free or reduced- 
price lunch 

Percent receives special education 

45 

45 

45 

0.715 

46 

46 

46 

0.675 

services 

16 

18 

14 

0.086 

17 

18 

15 

0.242 

Percent limited English proficient 

3 

4 

1 

0.741 

3 

4 

1 

0.247 

Percent female 

51 

50 

52 

0.478 

50 

50 

50 

0.765 

Percent racial/ethnic minority 

5 

6 

3 

0.577 

5 

7 

3 

0.659 

Mean grade 7 score on state 
mathematics assessment 

-0.19 

-0.21 

-0.18 


-0.23 

-0.24 

-0.22 


(standardized) 3 

(0.84) 

(0.83) 

(0.85) 

0.627 

(0.83) 

(0.83) 

(0.83) 

0.586 

Mean fall 2008 Promise 

313.81 

315.01 

312.57 


313.08 

314.30 

311.79 


Assessment pretest score b 

(26.83) 

(26.57) 

(27.07) 

0.330 

(27.05) 

(26.65) 

(27.43) 

0.407 

Total number of N-AR students' 1 

1,241 

630 

611 


1,349 

697 

652 



AR is algebra ready. N-AR is not algebra ready. 


Values are unadjusted. Differences in student characteristics by condition were tested using a model that accounts for the clustered 
data structure and blocking used for randomization. Figures in parentheses are standard deviations. 

a. State mathematics scores were standardized by using the mean and standard deviation of the test scores within each state, including 
only schools participating in the study. 


29 The rate of missing data on outcome measures was l%-3% for the AR sample and 7% — 14% for the N-AR 
sample. See appendix F for details about missing data and multiple imputation, and for a comparison of the 
characteristics of students with and without missing outcome data, see tables F-2 — F-5. 
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b. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

c. 434 AR students have algebra posttest data, and 427 AR students have coursetaking codes (indicators of advanced vs. not-advanced 
high school coursetaking as of spring of grade 9). Of the students with algebra posttest scores, 4 are missing free and reduced price 
lunch status, 3 are missing special education status, 3 are missing LEP status, 3 are missing state mathematics scores, 4 are missing 
study-administered pretest data, and 0 are missing gender or race/ethnicity data. Of the students with coursetaking codes, 3 are 
missing free and reduced price lunch status, 3 are missing special education status, 3 are missing LEP status, 3 are missing state 
mathematics scores, 3 are missing study-administered pretest data, and 0 are missing gender or race/ethnicity data. 

d. 1,241 N-AR students have complete algebra posttest data, and 1,349 N-AR students have coursetaking codes (indicators of 
intennediate vs. not-intennediate coursetaking as of spring of grade 8). Of the students with algebra scores, 3 1 are missing free and 
reduced price lunch status, 18 are missing special education status, 18 are missing LEP status, 29 are missing state mathematics 
scores, 38 are missing study -administered pretest data, 2 are missing gender data, and 2 are missing race/ethnicity data. Of the 
students with coursetaking codes, 37 are missing free and reduced price lunch status, 23 are missing special education status, 23 are 
missing LEP status, 37 are missing state mathematics scores, 47 are missing study-administered pretest data, 5 are missing gender 
data, and 6 are missing race/ethnicity data. 

Source : Maine state department of education and Vermont supervisory unions. 


Table 2-9. Baseline Characteristics of AR and N-AR Samples at Analysis (Missing Data 
Imputed) 


Characteristic 

Overall 

Treatment 

Control 

/j-value 

AR students 

Percent eligible for free or reduced-price lunch 

32 

34 

30 

0.605 

Percent receiving special education services 

4 

3 

4 

0.572 a 

Percent limited English proficient 

3 

5 

2 

0.646 a 

Percent female 

49 

49 

49 

0.833 

Percent racial/ethnic minority 

7 

8 

5 

0.975 

Mean grade 7 score on state mathematics 

0.90 

0.93 

0.88 

0.608 

assessment (standard deviation) 13 

(0.72) 

(0.63) 

(0.81) 


Mean fall 2008 Promise Assessment pretest score 

349.81 

353.55 

346.04 

0.038 

(standard deviation) 0 

(23.36) 

(22.33) 

(23.76) 


Total number of AR students 

440 

218 

222 


N-AR students 

Percent eligible for free or reduced-price lunch 

48 

48 

48 

0.863 

Percent receiving special education services 

18 

19 

17 

0.236 

Percent limited English proficient 

3 

5 

2 

0.989 a 

Percent female 

50 

49 

50 

0.718 

Percent racial/ethnic minority 

5 

7 

4 

0.592 a 

Mean grade 7 score on state mathematics 

-0.28 

-0.28 

-0.27 

0.872 

assessment (standard deviation) 13 

(0.91) 

(0.91) 

(0.91) 


Mean fall 2008 Promise Assessment pretest score 

311.81 

313.01 

310.54 

0.238 

(standard deviation) 0 

(27.67) 

(27.31) 

(28.00) 


Total number of N-AR students 

1,445 

744 

701 



AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control); 440 AR students (218 treatment, 222 control); and 1,445 N-AR 
students (744 treatment, 701 control); 4 control schools had no N-AR students. Values represent unadjusted means. Estimates 
were averaged across 10 multiply imputed datasets. Differences in characteristics by condition were tested using a model that 
accounts for the clustered data structure and blocking used for randomization. Figures in parentheses are standard deviations. 

a. Model does not estimate when controlling for five state by size dummy blocking variables. Reported p - value represents a 
model that controls for state and two dummy indicators for medium and large schools rather than their interactions. 

b. State assessment scores were standardized by using the mean and standard deviation of the test scores within each state, 
including only schools participating in the study. 

c. Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source : Maine state department of education and Vermont supervisory unions. 
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Tables 2-8 and 2-9 show that, as was the case at random assignment, differences between 
treatment and control groups were not statistically significant across measured demographic 
characteristics or prior mathematics achievement, except on the study administered pretest at the 
time of analysis. At analysis, AR students in treatment schools scored significantly higher than 
their counterparts in control schools (means = 353.45 and 346.04, respectively, for the AR 
treatment and control groups with complete algebra posttest data ,p = 0.039; means = 353.52 and 
346.16, respectively, for the AR treatment and control groups with complete coursetaking data,/? 
= 0.047; and means = 353.55 and 346.04, respectively, for the AR treatment and control groups 
with missing cases imputed.) 

Characteristics of Teachers in Study Schools 

There were no significant differences in the average number of mathematics teachers per school 
by condition (p = 0.997) (table 2-10). These numbers do not include the online teachers, who 
were additional to the treatment schools’ teaching staff (there was one online teacher per school). 
(Multiple schools shared a single online teacher.) Chapter 3 provides more infonnation on the 
allocation of online teachers to treatment schools. 


Table 2-10, Number of Grade 8 Mathematics Teachers (Excluding Online Teachers) 



Treatment schools 


Control schools 





(« = 35) 



(« = 33) 



State 

Total number 
of teachers 

Mean per 
school 

Standard 

deviation 

Total number 
of teachers 

Mean per 
school 

Standard 

deviation 

p-value 

Maine 

36 

1.44 

0.65 

39 

1.56 

0.82 

0.569 

V ermont 

16 

1.60 

1.35 

10 

1.25 

0.46 

0.495 

Total 

52 

1.49 

0.89 

49 

1.48 

0.76 

0.997 


Note : Tests of significance were conducted using two-tailed, independent sample t-tests. 
Source: Study records. 


Two-thirds of the schools had a single grade 8 mathematics teacher, and 24 percent had two 
grade 8 mathematics teachers; in 90% of schools, a primary mathematics teacher provided 
instruction to grade 8 students and a second teacher provided instruction to particular groups of 
students (special education, gifted and talented). The remaining 10% of schools had three, four, 
or five grade 8 mathematics teachers (table 2-11). 
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Table 2-11. Percentage of Participating Schools with Various Numbers of Grade 8 
Mathematics Teachers (Excluding Online Teachers) 


Number of grade 8 
mathematics 
teachers in school 

Overall 

Treatment 

Control 

/i-value 

Number 

Percent 

Number 

Percent 

Number 

Percent 

1 

45 

66 

24 

69 

21 

64 

0.867 

2 

16 

24 

7 

20 

9 

27 

0.201 

3 or more 

7 

10 

4 

11 

3 

9 

0.940 

Total 

68 

100 

35 

100 

33 

100 

— 


— is not appropriate to calculate because schools were randomly assigned to treatment or control. 

Note: Sample includes 68 schools (35 treatment, 33 control). Tests of significance were conducted using two-tailed jl tests. 
Schools with three, four, and five mathematics teachers were collapsed into one group to protect the confidentiality of 
schools. Totals may not sum to 100 percent because of rounding. 

Source: Study team calculations based on school rosters and teacher survey records. 


Table 2-12 summarizes the background characteristics of the teachers in study schools, including 
online teachers and grade 8 mathematics teachers in both treatment and control schools. The 
background characteristics are based on infonnation collected from the teacher survey conducted 
in spring 2009. These data indicate that the regular grade 8 mathematics teachers at treatment 
and control schools were similar in terms of education, gender, race/ethnicity, number of years 
teaching, and certification to teach secondary mathematics. More online teachers than teachers 
in the treatment or control schools were certified to teach secondary mathematics and had 
master’s degrees or higher degrees (approximately 75% of online teachers vs. approximately 
38% of regular teachers in treatment schools and approximately 20% of regular teachers in 
control schools). Online teachers also had significantly more experience teaching algebra than 
classroom teachers (9.5 years for online teachers versus 2.3 years for regular teachers in 
treatment schools and 3.7 years for regular teachers in control schools). There were no 
differences in the total number of years teaching or the number of years spent teaching middle 
school mathematics. 


30 Certification data are based on teachers’ self-reports of certification status. 
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Table 2-12. Background Characteristics of Online and Regular Grade 8 Mathematics Teachers 



Online 









Regular teachers in 


teachers 

Regular teachers 

Regular teachers 

Online teachers vs. 

Online teachers vs. 

treatment schools vs. 


(all in treatment 

in treatment 

in control 

Regular teachers in 

Regular teachers in 

Regular teachers in 


schools) 

schools 

schools 

treatment schools 

control schools 

control schools 








Percent 


Percent 


Percent 


Characteristic 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Difference 

/7-value 

Difference 

/;-value 

Difference 

/j-value 

Certified in secondary 
mathematics 

8 

100 

22 

39 

20 

37 

61 

0.003 

63 

0.003 

2 

0.959 

Master’s degree or 
above 

t 

t 

21 

38 

11 

20 

t 

0.017 

t 

0.001 

18 

0.060 

Female 

t 

t 

27 

48 

26 

48 

t 

0.236 

t 

0.087 

0 

0.988 

White 

t 

t 

44 

79 

43 

80 

t 

0.789 

t 

0.851 

1 

1.000 



Standard 


Standard 


Standard 

Mean 


Mean 


Mean 



Mean 

deviation 

Mean 

deviation 

Mean 

deviation 

Difference 

/7-value 

Difference 

/7-value 

Difference 

/7-value 

Years of teaching 
experience (total) 

22.87 

13.15 

14.8 

9.17 

14.69 

9.77 

8.07 

0.034 

8.18 

0.043 

0.11 

0.955 

Years teaching middle 
school mathematics 

6 

8.47 

9.78 

7.93 

10.64 

OO 

oo 

-3.78 

0.220 

-4.64 

0.173 

-0.86 

0.616 

Years teaching 
Algebra I 

9.5 

6.48 

2.29 

3.67 

3.71 

5.25 

7.21 

0.001 

3.13 

0.008 

-1.42 

0.135 


Note : Sample includes 68 schools and 117 teachers (8 online teachers, 56 regular teachers in treatment schools, and 53 regular teachers in control schools). Survey data were 
missing for 14 teachers. The difference between two groups is indicated by (y — x). Tests of significance for categorical variables were conducted using two-tailed, y2 tests. Tests 
of significance for continuous variables were conducted using two-tailed, ANOVA models. To maintain the confidentiality of participants, measures with fewer than three 
missing data cases were suppressed for presentation purposes. 

{ To maintain confidentiality of participants, the number was suppressed for presentation purposes, 
f To maintain confidentiality of participants, the percentage was suppressed for presentation purposes. 

Source: Teacher survey. 




Characteristics of High Schools Attended by AR Students 

To collect follow-up information on the mathematics courses-taking in high school by AR 
students, at the end of grade 8 researchers gathered information about which high school each 
AR student was planning to attend the following year. When the students were in grade 9, 
researchers contacted the high schools to confirm the enrollment of each student. Sixty-six high 
schools received AR students (45 public schools and 21 private schools). The high schools 
attended were in Maine (43 schools), Vermont (16 schools), and other states (7 schools). The 
number of AR students enrolled in each high school varied from fewer than 4 to 29. The 
characteristics of the high schools attended by AR students are shown in table 2-13. 

Of the 66 high schools, 41% received AR students from treatment schools only, 36% received 
AR students from control schools only, and 23% received AR students from a both treatment and 
control schools. Within the high schools that received students from both conditions, 45% of the 
AR students were from treatment schools and 55% were from control schools. 


Table 2-13. Characteristics of High Schools That Received AR Students in 2008/09 


Characteristic 

Number 

Mean 

Standard 

deviation 

Eligible for free or reduced-price lunch (average percent) 3 
Race/ethnicity (average percent) 

43 

34 

13.66 

White 

63 

89 

14.91 

Black 

63 

2 

3.73 

Hispanic 

63 

2 

1.43 

Asian/Pacific Islander 

63 

4 

6.16 

Native American 

63 

3 

12.01 

Female (average percent) 

63 

49 

8.63 

Proficient on state mathematics assessment (average percent) 

47 

35 

13.67 

Student-teacher ratio 

63 

10 

2.63 

Number of grade 9 students 

62 

114 

79.75 

Total school enrollment 

63 

502 

311.42 


Note: Sample includes 66 high schools; 45 (68%) were public schools and 21 (32%) were private schools. No data were 
available for three private high schools. 


Free or reduced-price lunch data were unavailable for any of the 21 private high schools because they do not participate 
in the National School Lunch Program. 

Source: Data on public high schools are from the Common Core of Data (2007/08). Data on private schools are from the 
Private School Universe Survey (2007/08). 


Estimation Methods 

All analyses were conducted separately on the AR and N-AR student samples. The samples used 
for analyses are intent-to-treat samples, meaning that all students with consent who were 
identified as AR before random assignment were included in the AR student sample, whether or 
not they enrolled in or stuck with the online Algebra I course. 

Consistent with the research questions that frame the study, the two primary analyses assess the 
impact of online Algebra I on the algebra test scores of AR students and their likelihood of 
participating in an advanced course sequence in high school. 
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Four secondary analyses assess the impact of online Algebra I on: 

• AR students’ general mathematics scores 

• N-AR students’ algebra scores 

• N-AR students’ general mathematics scores 

• N-AR students’ likelihood of participating in an intermediate course sequence in high 
school (based on planned grade 9 courses). 

Given the nested structure of the data (the clustering of students within schools), multilevel 
models were used to estimate the impacts of online Algebra I on the primary and secondary 
study outcomes. For each student-level outcome, two-level hierarchical models were used, with 
students nested within schools. 

To increase the precision of the estimates in these analyses, researchers used a set of baseline 
characteristics of schools and students as covariates. The following covariates were included in 
the outcome analyses: 

• School level (blocking variables) 

o State 

o School size (based on the number of grade 8 students enrolled in spring 2008) 

• Student level 

o Baseline mathematics achievement scores on the state mathematics assessment 
from the year before the study 

o Student-level demographic information (gender, eligibility for free or reduced- 
price lunch, special education status). 

In addition to estimating the main impact models, researchers also conducted sensitivity 
analyses. Six sensitivity analyses were conducted to test the robustness of the observed estimates 
from the benchmark impact model on AR students’ algebra scores; another six sensitivity 
analyses tested the robustness of the high school coursetaking outcome. These analyses tested 
models with different methods of accounting for students’ baseline mathematics achievement, a 
model with no covariates, a model based on observed (nonimputed) data only, and a model that 
tested an alternative nesting structure (with students in treatment schools clustered within online 
teachers). 

Researchers also conducted 10 sensitivity analyses to test the robustness of the findings for N- 
AR students’ outcomes. For each outcome, these analyses test three models with different 
methods of accounting for students’ baseline mathematics achievement, a model with no 
covariates and a model based on observed (nonimputed) data only. 

Treatment of Missing Data 

Rates of missing data on baseline covariates and outcome measures were 1% - 3% for the AR 
sample and less than 1% - 14% for the N-AR sample (see table F-l in appendix F). The highest 
rates of missing data for the N-AR student sample were on the algebra posttest. On this measure, 
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14% of the sample was missing because of absence or attrition (6%) or because the score was 
eliminated because the student spent less than five minutes on the test (8%) (see table F-l in 
appendix F). 

To handle missing data, researchers used multiple imputation by chained equations. Multiple 
imputation models were specified on the basis of the analysis of predictors of missingness; they 
included student and school covariates and interaction terms between student covariates. Within 
both the AR and N-AR samples, multiple imputation was conducted for the treatment and 
control groups separately. Appendix E provides the results of sensitivity analyses testing whether 
results were affected by the imputation. Appendix F provides more information on missing data 
patterns and the multiple imputation procedure. 

Statistical Precision and Significance Testing 

Two-tailed /-tests were used to assess the statistical significance of the average impact estimates. 
If an impact estimate was statistically significant, it is possible to conclude with some confidence 
that the online Algebra I course had an effect on the outcome being assessed. If an impact 
estimate was not statistically significant, the nonzero estimate may be a product of chance. 
Statistical significance does not capture the magnitude or meaning of an impact estimate, only 
the probability that an effect of the size observed might occur if the true impact were zero. 
Statistically significant impacts may or may not be policy relevant. 

Multiple Hypothesis Testing 

This study takes different approaches to multiple hypothesis testing for the primary and 
secondary analyses, because the framework for determining whether the intervention was 
“successful” was based on criteria that establish different types of hypotheses. The criteria, stated 
in chapter 1, are repeated here: 

• A statistically significant positive impact on either of the two primary research 
questions (AR students’ algebra scores at the end of grade 8 or high school 
coursetaking). 

• The absence of statistically significant negative side effects (as assessed by the four 
secondary research questions). 32 


For the primary analyses, intervention success is achieved if a positive impact on either of the 
two primary outcomes is detected (criterion 1). For this analysis, an adjustment was applied to 
account for multiple comparisons. Specifically, to maintain the probability of falsely detecting a 
statistically significant result {p < .05) if there were no true impact on either of the two primary 


31 In addition to this model, the study team explored a second imputation model that included school-level fixed 
effects (dummy variables for each school). However, because of the large number of school dummy variables needed (68) 
and the presence of low-incidence binary variables (such as racial/ethnic minority status in this sample), the fixed-effect 
approach was inappropriate method for use with the sample (see appendix F). 

32 For the secondary research questions, the study was not designed to determine whether the groups were 
statistically equivalent. A lack of statistical significance for an impact estimate does not mean that the impact being 
estimated equals zero. Rather, it means that the estimate cannot reliably be distinguished from zero, an outcome that 
may reflect the small magnitude of the impact estimate, the limited statistical power of the study, or both. 
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outcomes, the study team adjusted the statistical significance level for each of the two primary 
outcomes to 2 . 5 % (Bonferroni correction). For primary analyses in this report, statistical 
significance is denoted in the tables by an asterisk when the /7-value of the impact estimate is less 
than or equal to 0 . 025 . 

For the four secondary analyses, adjustments for multiple comparisons were not applied. Making 
such adjustments would have made the tests less conservative, because on these questions, a 
desirable outcome in favor of the intervention is a lack of statistically significant side effects. 

Had the statistical significance level for these tests been reduced, it would have been easier to 
conclude that there were no detectable side effects than if the level at 5 % for each test had been 
maintained. The study team thus opted for the more conservative approach of maintaining 5 % 
significance for each test. Multiple comparisons adjustments were not applied for the two 
exploratory impact analyses, which are reported in chapter 5 . For secondary analyses, statistical 
significance is denoted in the tables by an asterisk when the /7-value of the impact estimate is less 
than or equal to 0 . 05 . 
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Chapter 3 Implementation of the Intervention 

The purpose of this chapter is twofold: it describes the online Algebra I course intervention and 
how it was implemented in the study and describes the regular grade 8 mathematics instruction 
provided in treatment and control schools. The chapter begins with an overview of the 
responsibilities of the participating schools and the online course provider and a description of 
the fonnation of the online Algebra I classes. 

The rest of the chapter describes mathematics instruction for AR and N-AR students in treatment 
schools and control schools in terms of the six programmatic features of the intervention defined 
in Chapter 1 . The first section details the implementation of the online course in treatment 
schools with a focus on the mode of instruction and course content. The second section describes 
the mode of instruction and content of the mathematics courses provided to N-AR students in 
treatment schools (that is, students who did not take the online course), and of the mathematics 
courses provided to AR and N-AR students in control schools (who did not have access to an 
online Algebra I course). The third section describes and compares the mathematics instruction 
delivered to AR and N-AR students in treatment and control schools in terms of teacher 
qualifications, staffing intensity, class size, and ability grouping. 

Overview of the Intervention: Course Provider and School Responsibilities 

Class, com Responsibilities 

A variety of online learning models are available. One option is for a school to purchase a “seat” 
in an online course from an online course provider for each student it wants to enroll. 33 In this 
model, the online course provider is responsible for overseeing the operation of the online 
course, including hiring, training, and supervising the online teacher. Based on the constraints 
that rural schools may encounter that prevent them from offering a face-to-face online Algebra I 
course — such as the lack of a teacher qualified to teach Algebra I or the small number of AR 
students — REL-NEI determined that this was an appropriate model to test for the study. 

As described in chapter 1, the study team selected Class.com to deliver the yearlong online 
Algebra I course that was the intervention for the study. Class.com provided a senior 
mathematics specialist whose role involved hiring and supervising the online teachers and 
ensuring that the course curriculum was taught as intended. 

In June 2008, Class.com hired eight teachers from its network of mathematics teachers to serve 
as online teachers for the study. All teachers were certified to teach mathematics and met both 
states’ “highly qualified teacher” content knowledge criteria. 34 

All online teachers attended a two-day training workshop at Class.com headquarters about a 
month before the beginning of the school year; six teachers also attended an optional one-day 


33 As of the 2010/1 1 academic year, the cost of purchasing a seat in one of Class. corn’s Algebra I courses was $700 
a year. 

4 According to the No Child Left Behind Act (NCLB), a “highly qualified” secondary teacher must demonstrate 
subject matter competence in each subject he or she teaches (see Title IX, Part A, Section 9101 (23) for a complete 
definition of “highly qualified” ). The online teachers met Maine and Vermont’s content knowledge criteria for 
being highly qualified; they were not required to be certified in either state. 
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workshop in January. The purpose of the summer training was to familiarize the teachers with 
the structure and operation of the online course, demonstrate how to operate the courseware and 
use the embedded communication tools, and suggest methods for guiding student progress. The 
January workshop reviewed the topics presented in the summer workshop. 

Class.com also provided a senior mathematics specialist to oversee the administration of the 
online course. This specialist was a Class.com employee who had helped develop the Algebra I 
course and who had a background in mathematics education. Class.com hosted the online course 
on its servers, assigned students (by school) and teachers to online course sections, and was 
responsible for ensuring that the online course was in place and running continuously in all 
treatment schools throughout the school year, regardless of the schools’ start and end dates for 
the academic year. According to archival data from the Class.com management system, as well 
as study team communication with participating schools, Class.com fulfilled these requirements, 
and the course was delivered without interruption during the 2008/09 school year. No reports or 
observations were made of any Class. corn-related technical problems interfering with students’ 
access to the course during the school year. 

School Responsibilities 

During recruitment, the study team informed all schools that if assigned to the treatment group, 
their implementation of the online course would have to meet four requirements. 

1 . Schools were required to provide each AR student with in-school access to a computer 
with a high-speed Internet connection and the appropriate web browser application. 

2. Schools were to assign AR students to a regular class period during which they would 
access the online course. This period needed to meet with the same frequency and 
duration as the school’s regular grade 8 mathematics classes. 

3. Schools were required to provide a school staff member to serve as a proctor who would 
supervise and support students while they were using the online course. The proctor did 
not have to be a mathematics teacher and was not required to provide instruction. 

4. The online Algebra I course had to serve as AR students’ only grade 8 mathematics 
course. 

All of the treatment schools fulfilled these requirements. 

Formation of the Online Algebra I Classes 

As noted in chapter 2, 242 students in the 35 treatment schools were assigned to 10 online 
sections. Of these course participants, 211 (87%) students were identified by their schools before 
random assignment as algebra ready. In the fall of the study year, schools were allowed to 
identify additional students for the course. A total of 3 1 N-AR students (4% of the 782 N-AR 
students in treatment schools) enrolled in the course. 


35 There were reports of brief, periodic interruptions in Internet access in a small number of schools, and a few 
schools experienced a delay in assigning computers to students. 
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Each online teacher taught one or two sections, with an average of 24 students per section (table 
3-1). Eight of the 10 sections included students from multiple schools, but all the online course 
sections were composed of students from either Maine or Vermont — no section included 
students from both states. There were 7 online sections with 173 students from Maine and 3 
sections with 69 students from Vermont. These sections were formed by the study’s director of 
implementation and Class.com, primarily by grouping schools that had a similar amount of time 
per day for grade 8 mathematics (for example, 50 minutes every day versus a double block three 
times a week) and other scheduling similarities. 


Table 3-1. Numbers of Students and Schools per Online Course Section 


Section 

Number of 
schools 

Number of AR 
students 

Number of N- 
AR 

students 

Total number 
of students 

Teacher 

1 

4 

13 

0 

13 

E 

2 or 3 

5 

36 

11 

47 

C and G 

4 

5 

25 

5 

30 

G 

5 or 6 

7 

42 

0 

42 

B and D 

7 or 10 a 

8 

40 

9 

49 

H and F 

8 or 9 a 

6 

55 

6 

61 

A 

Total 

35 

211 

31 

242 

8 


AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 35 schools and 242 students. Because six students in section 3 did not receive their computers 
until several weeks into the academic year, Class.com originally assigned them to a separate course section taught by 
teacher G. After several months, teacher G requested that these students be merged into an existing section (section 3) 
because it was more efficient to supervise the work of one larger section than two smaller sections. Therefore, the 1 1 
sections collapsed to 10 sections midway through the school year. 

a. Sections with fewer than four schools or students were aggregated to protect the confidentiality of participating 
schools and students. 

Source: Study team records and online course activity archival data. 

Mode of Instruction and Content of the Online Algebra I Course Offered to AR 
Students in Treatment Schools 

The following sections describe the implementation of the online Algebra I course in treatment 
schools in terms of the mode of instruction (including fidelity of implementation) and the content 
of the course (including coverage and course completion rates). 

Mode of Instruction 

Mathematics instruction for AR students in treatment schools had three primary components: the 
online Algebra I course, the online teacher, and the on-site proctor. 

Unlike a traditional course, in which the primary mode of instruction is for a teacher to provide 
in formation directly to students, the primary mode of instruction for students taking the online 
Algebra I course was their interaction with the materials and activities available online. 

Class.com delivered the online course activities through Moodle, an open source course 
management system. 

The online course presents material as units, lessons, and topics. Each topic is presented to 
students in the form of an electronic, interactive textbook. The textbook includes computerized 
direct instruction; guided practice (“your-tum” problems) and practice problem sets, both with 
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automated feedback; and quizzes and exams that provide immediate scores. Other activities 
include demonstrations of content materials, audio clips, interactive applets that present 
questions and guided solutions, a messaging feature through which students can send and receive 
messages from the online teachers, and a discussion board to which students can post questions 
or comments. 

Each lesson within a unit is divided into sets of topics that are designed to be completed within a 
50- to 55-minute period. The following are examples of typical mathematics activities that 
students encountered daily. The examples are drawn from the Slope-Intercept topic of the Other 
Forms of Linear Equations lesson in the Linear Equations unit (figure 3-1). 

• The topic begins with static text, which students read. Rollover definitions are available 
for important tenns. These terms appear in bold, underlined, blue type; the definition 
appears near the term when users place the cursor over the term (figure 3-2). 

• As students progress through a topic, the material they encounter alternates between 
static text and “chalktalks,” short videos with audio that serve as mini-lessons. The audio 
portion of the chalktalks is a voiceover describing the solution steps of the problem 
presented (figure 3-3). 

• Computer-scored “your-tum” problems and practice set problems are a primary 
component of the course. Students receive immediate feedback on whether their answers 
are correct or incorrect by selecting “Check My Answers.” Figure 3-4 presents practice 
set problems. It shows how the scoring appears to students when they check their answers 
and how an explanation is presented when they select “please explain.” 

• Interactive activities are sometimes paired with problem sets or content pages. The 
complexity and instructional purpose of the interactive applets vary and can consist of an 
interactive demonstration, guided questions, or open-ended prompts. In the interactive 
activity within this topic, students must graph two points to create a line before 
proceeding with the first set of practice problems (figure 3-5). 

• Quizzes (both practice and graded) are included at the end of each lesson; exams 
(practice and graded) are given at the end of each unit. Quizzes and exams consist of item 
sets randomly generated by the course management system from Class. corn’s item banks. 


36 It is the responsibility of students to complete the problems, check their work, and follow up if they do not 
understand something; the online course management system does not record whether students attempt the practice 
problems or record their scores. 
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Figure 3-1. Unit Menu: Lessons and Topics 



Unit 1 / Lesson 3 / Topic 2 


Page 1 


Topic 6: Graphing Applications: Distance, Rate, and ... 
Topic 7: Graphing Applications: Fixed and Variable C .. 
Lesson Review 


Quiz 

Lesson 3: Other Forms of Linear Equations 
Introduction 
Topic 1 : Slope of a Line 
Topic 2: Slope-Intercept Form 
Topic 3: Writing the Equations of Lines 
Discussion Group: Describing Linear Equations 
Topic 4: Line Relationships 
Topic S: Interpreting Scatter Plots Correlation 
Topic 6: Line of Best Fit 
Lesson Review 
Quiz 


Figure 3-2. Static Text with Hyperlinked Terms and Definitions 



Slope-Intercept Form 


T 


Page 1 


l 


Have you noticed in the graph of a linear equation (a straight line) that if the line 
is not vertical, there is a point where the line crosses the y-axis? This point is 
called the v-intercept and is a very important point in the graphing of a line. 
There is also a corresponding point on the x-axis called the x-intercept , where 
the graph of a line that is not horizontal will cross the x-axis. Let us look at a 
couple of lines and note these two points. 

Let's look at some examples and then you can try one on your 


The x-intercept of a line is the point 
where the line crosses the x-axis. It 
is found algebraically by substituting 
y = 0 into the equation. 


Figure 3-3. “Chalktalk” Short Videos 
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Figure 3-4. Practice Set Problems and “Check My Answers” Feature 


Page 1 


1. On line c£j, the y-intercept is 

2. The ordered pair for the y-intercept is ( 

3. The x-intercept is 

4. The ordered pair for the x-intercept is ( 


, ✓✓ 

[c#r«*e#r My Answer* 



Unit 1 / Lesson 3 / Topic 2 


T 


1. On line CD , the y-intercept is 4 


vO 

[Please Explain] 


2. The ordered pair for the y-intercept is ( 0 

X 

3. The x-intercept is 2.5 


4. The ordered pair for the x-intercept is (2.5 , 0 



, // 

[Check My Answers ] 


Page 1 


1 


ROO Answer Window 

,Ql http://content.class.com/c3_content/lor/mathematiri4' \ 


The line crosses the x-axis at 2. Therefore, the x-intercept = 2. 


Figure 3-5. “Your Turn” Interactive Exercises 


Tools 

vour Turn ■ 


Unit 1 / Lesson 3 / Topic 2 


Page 1 


Draw the graph of the line containing the following points: C (4, - 4) and D (1, 2). 


Move your mouse over the graph and 
plot each coordinate from the table by 
using the red circular target and 
clicking on the desired coordinate. 










y 







j 


£ Reset } 
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For each topic, students typically encounter an average of 7 web pages of static and interactive 
text, 1 or more chalktalks, 1 or more interactive activities, 8 “your-tum” problems, and 10 
practice set problems. Topics conclude with a summary of key ideas presented and practiced 
during the topic set. The Moodle online course management system gives students the means to 
communicate with the online teacher (described in more detail in the following subsection) and 
to access information on their grades, including the cumulative percentage correct and the 
percentage of correct answers on all quizzes and exams, and copies of all their online 
communications with the online teacher. 

To engage in the online course’s instructional activities, students need to log in to the online 
course. As described in chapter 2, student login activity is archived by the course management 
system and analyzed to describe students’ and online teachers’ activity in the online course. The 
study team observed the online course instruction and interactions between online teachers and 
their students using archived data provided by the online course provider. Specifically, for each 
of the 10 online course sections, the study team randomly selected one school day each month 
(October-May) and downloaded all online activity over a 24-hour period. Researchers collected 
24-hour samples of archived data once a month for eight months for each of the 10 course 
sections, for a total of 80 observations (eight per section). The analysis of these data shows that 
on average, 75% of students in each online course section logged into the course during the 24- 
hour observations. Students were expected to log into the online course at least once during their 
daily mathematics practice. 

The Role of the Online Teacher 

This section describes the expectations of the online teachers and presents data describing their 
actual activities during the course of the study year. According to Class.com, the role of online 
teachers is to grade written assignments, review students’ scores on quizzes and exams, coach 
and motivate students, conduct online discussions, and demonstrate concepts and processes. 
Online teachers work asynchronously with students, meaning that they are typically not online at 
the same time as students. During the two-day training workshop for the online teachers, 
Class.com demonstrated the means through which teachers could use the online course 
management system to communicate with students and monitor their progress in the course. In 
the training, Class.com indicated that online teachers should monitor students’ progress on a 
daily basis and communicate with students on most days. 

Monitoring Progress. Class. corn’s stated expectation for the online teachers is that they log in 
to the course management system daily to monitor student progress. Although the course 
management system gives teachers several types of information on student performance and 
progress through the course, some aspects of course progression are not accessible to teachers. 
Teachers can see the unit on which students are working, but they do not have access to 
information on where a student is within the unit (that is, which lesson or topic). Teachers do not 
have information on students’ completion of or scores on problem sets within lesson topics. 
Teachers do have a record of students’ scores (percent correct) on graded quizzes and exams, and 


’ 7 The rate at which students did not log on to the course during the observation period is in line with anecdotal 
reports from Class.com that students missed a number of days, particularly in the second half of the year, because of 
grade- or schoolwide activities. 
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they can see which items from the test ha nk were included in quizzes and exams and which 
problems students answered correctly and incorrectly on quizzes and exams. 

To assess the regularity with which online teachers logged in to the course to monitor student 
activity, the study team analyzed the online course activity archived by the course management 
system and tallied whether each section’s online teacher logged on to the course management 
system at least once during each 24-hour observation period. On average, online teachers logged 
in during 96% of the observation periods, suggesting that they were logging on to the course 
daily. The study team also examined the 24-hour observations to record teachers’ activities while 
online. Although online teachers were provided explicit guidance to monitor student login and 
general course activity on a daily basis, section teachers monitored student login activity at least 
once during the observation period in only 70% of the observed sessions. Online teachers were 
not provided explicit guidance about the frequency with which they should examine student 
grades (by going to the “student gradebook”). They were observed to do so in 43% of the 
observation periods. 

Communicating with Students. Online teachers are expected to read and respond to student 
messages daily and to send messages (through the course management system) to all students 
daily as they review student work. Online teachers can communicate with students by using the 
online messaging system available through Moodle and review records of these 
communications. In addition to sending messages to students, online teachers are expected to 
reply to any student-initiated messages within 24 hours to ensure that students receive the reply 
the next time they are scheduled to log in to the online Algebra I course. Online teachers can also 
use the course home page to make announcements to students in their sections or post additional 
materials, such as review sheets or additional work to support student mastery of a particular 
concept. 

To assess the frequency and regularity of teacher communications with individual students, 
researchers counted the number of teacher-to-student messages and then coded the content of the 
message and identified whether it was course related. Messages were coded as providing 
administrative feedback (for example, grades, the pace at which a student was progressing 
through the course) or mathematics content (for example, encouraging understanding reflection 
or critical thinking; providing constructive feedback; and using incorrect answers as learning 
opportunities. The number of messages are assumed to be related to the number of students in 
each section. Therefore, rather than report the number of online messages sent during the 
observations, researchers first reported the percentage of observations during which teachers sent 
at least one message and then reported the average percentage of students in the section who 
received messages. 

Online teachers sent at least one message to a student in 91% of the observations; on average, 
27% of students in each section were recipients of these messages. Messages included 
administrative feedback, mathematics content feedback, or other content, such as greetings. 


38 All online communication between students and teachers occurred only through the online course’s messaging 
system. 
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Online teachers sent at least one message to a student that contained administrative feedback in 
51% of the observations (an average of 4.1 of the eight observations per section). On average, 
teachers directed this feedback to 13% of the students in the section. 

Online teachers sent at least one message to a student that contained mathematics content in 18% 
of the observations (an average of 1.4 of the eight observations per section). On average, teachers 
directed these messages to 3% of the students in the section. 

Students could also initiate communication with their online teacher. At least one student asked 
the online teacher a question through the course messaging system in 81% of the observed 
sessions. On average, 10% of students in a section sent a message to their teacher with a 
question. In observations where students asked questions, the online teacher replied to this 
question within 24 hours an average of 96% of the time. 

In summary, online teachers logged in to the course at least once a day to monitor students’ 
activity or progress, and they communicated directly with approximately one-fourth of students 
every day. Communications with explicit mathematics content were infrequent (more frequent 
were messages containing administrative feedback). When a student contacted the online teacher 
directly, the teacher almost always replied within 24 hours. 

The Role of the On-Site Proctor 

Every treatment school provided an on-site proctor, who was available to students during the 
designated class periods. The role of the proctor was to ensure that students had access to the 
required technology, proctor exams, supervise students’ behavior, serve as a personal contact for 
students and parents, and serve as the liaison between the online teacher and the school or 
parents. Because their role did not include providing mathematics instruction, proctors did not 
need to be qualified mathematics teachers. The schools selected the staff members who served as 
proctors. 

Before the beginning of the 2008/09 school year, the proctor and a technical support staff 
member from each treatment school participated in a one-day training. Class.com staff delivered 
the training sessions in three locations (two in Maine and one in Vermont). The six-hour training 
sessions covered the structure of the online course and provided hands-on training on operating 
the courseware, including viewing students’ progress through the course, accessing their 
assessment scores, and using the embedded communication tools. The training also suggested 
methods for helping students keep track of their own progress. All 35 treatment schools sent at 
least one representative to the training, with 33 schools sending their proctor and 27 schools 
sending a technology support staff member. 

Data from the weekly proctor logs indicated that in 28 of 35 treatment schools (80%), the proctor 
was the grade 8 mathematics teacher. In the remaining schools, the proctors were other 
mathematics teachers in the school, the principal, or the education technology specialist. 

In more than 90% of treatment schools, proctors had other responsibilities during the time they 
were supervising the online students. In 69% of schools (24 of 35), proctors were teaching 
another class while proctoring the online course). In other schools, students accessed the course 

39 Some schools had more than one proctor. 
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from their “own” separate classroom, the library, the computer lab, or a combination of a 
classroom and another space in the school. 

Although the role of the proctor as defined for the study did not include providing mathematics 
instruction, the proctors (80% of whom were also grade 8 mathematics teachers) were a source 
of instructional support for online students. Proctors’ weekly logs reveal that they spent an 
average of about 50 minutes a week answering students’ algebra-related questions and 10-14 
minutes a week answering nonalgebra mathematics questions (table 3-2). They also assisted 
students with technical issues and communicated with the online teachers. 

Table 3-2. Proctor Activities Associated with the Online Course 

Fall Winter Spring 

(November- (January- (April - 
December March June 

Activity 2008) 2009) 2009) 

Minutes per week 

Answering students’ algebra-related questions 

Answering students’ nonalgebra mathematics 
questions 

Providing students with technical support 
Communicating with online teacher 

Communicating with Class.com technical support 

Communicating with in-school technical support 
staff 

Note: Sample includes 35 schools and 38 proctors. Figures in parentheses are standard deviations. 

Source: Proctor logs. 


54.2 

53.6 

50.1 

( 44 . 1 ) 

( 55 . 8 ) 

( 54 . 6 ) 

12.8 

15.5 

12.3 

( 14 . 7 ) 

( 28 . 6 ) 

( 28 . 0 ) 

10.2 

7.9 

8.0 

( 17 . 0 ) 

( 9 . 7 ) 

( 13 . 3 ) 

5.2 

6.2 

6.2 

( 4 . 9 ) 

( 4 . 0 ) 

( 4 . 9 ) 

0.8 

0.7 

1.0 

( 3 . 3 ) 

( 2 . 6 ) 

( T 9 ) 

2.1 

1.5 

2.5 

( 3 . 4 ) 

( 2 . 2 ) 

( 5 . 8 ) 


Course Content 

The Class.com course is divided into two parts, Algebra 1A and Algebra IB, with each part 
designed to be equivalent to a semester in a traditional middle or high school Algebra I course. 
Algebra 1 A has five units, which address symbols and number properties, functions and 
equations, equations and problem solving, inequalities and absolute value, and polynomials. 
Algebra IB has four core units, which focus on functions and relations, systems of equations and 
inequalities, the simplification of rational and radical expressions, and quadratic equations. Two 
additional units focus on statistics and probability. 40 

According to Class.com, students need 32-34 weeks (160-170 days) to complete Algebra IA and 
IB, assuming 40-50 minutes of instruction each day. Both Maine and Vermont mandate a 175- 
day school year for grade 8 students, which is sufficient time to complete Algebra IA and IB 
according to the Class.com schedule. 


40 Appendix C provides detailed information on the content of each portion of the course. 
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In both states, state standards for grade 8 mathematics include algebraic concepts that overlap 
with many of the topics in the Class.com curriculum; however, neither state has specific content 
standards for Algebra I. The study team compared the mathematics content in the Class.com 
course with the content of two Algebra I textbooks in use in Maine and Vermont. 41 As described 
in appendix C, the Class.com course was well aligned with these curricula. Additional 
information on the structure and content of the online course is provided in appendix C. 

Online Course Completion Rates 

An important aspect of the course content in the intervention is the amount of the course students 
actually completed. The course offered by Class.com is designed as a full-year course. The 
expectation is that students can move slightly ahead or spend extra time covering topics if 
necessary, but the course is not intended to be self-paced. 

The pace at which students actually progressed varied, as did rates of course completion. About 
43% of AR students completed all nine units of Algebra IA and IB, and another 39.3% 
completed all of IA and some of IB (six, seven, or eight units) (table 3-3). A student 
“completed” a unit by passing the respective end-of-unit test with a score of 60% or higher. The 
study team defined passing as 60% or higher because the standard criterion for “passing” varied 
across participating schools, some setting the passing criterion at 60% and others at 70%. The 
60% threshold was chosen so that students would not be held to higher standards by the course 
than was typical for their school. The only significant predictor of course completion was 
baseline mathematics achievement, as measured by the study-administered Promise Assessment: 
students who scored above the mean on the study-administered pretest were more likely to 
complete the online algebra course than student who scored below the mean (table 3-4). 

Table 3-3. Number of Online Algebra I Course Units Completed by AR Students 


Portion of Online Algebra I 

Number of units 

Percentage of 

Course Completed 

completed 

students 

Some but not all of IA 

Fewer than 5 

1 1 .9 a 

All of I A 

5 

6.2 

IA and some but not all of IB 

6 

16.6 


7 

11.8 


8 

10.9 

All of IA and IB 

9 

42. 6 b 


Note: Sample includes 35 schools and 21 1 AR students. Algebra IA has five core units and Algebra IB has four core units. 
Students who completed nine core units completed Algebra LA and Algebra IB. There was variation in how much of the course 
students completed within and across the 10 online sections. 

a. Number of students who completed fewer than five course units included 9 (4.3%) of AR students who withdrew from the 
online course. 

b. Thirty-one percent of the 211 AR students also completed the two additional units on statistics and probability beyond 
Algebra IA and IB. 

Source: Study team records based on online course activity archival data. 


41 The two textbooks are Algebra: University of Chicago School Mathematics Project (McConnell, 2002) and 
Glencoe Algebra 1 (Holliday et al. 2005). 

42 Thirty-one N-AR students were also enrolled in the on-line course in treatment schools, however because all 
impact analyses take an intent-to-treat rather than a treatment-on-the-treated approach, these students were excluded 
from descriptive statistics of course completion and inferential statistical analyses. 
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Table 3-4. Online Course Completion Rates, by Student Characteristics 


Characteristic 

Total 

students 

Number 

completing 

course 

Percent 

completing 

course 

/i-value 

Eligible for free or reduced-price lunch, 
special education services, or limited English 
proficiency 3 

78 

28 

36 

0.061 

Not eligible for free or reduced-price lunch, 
special education services, or limited English 
proficiency 

131 

62 

47 

Female 

102 

46 

45 

0.345 

Male 

108 

44 

41 

Racial/ethnic minority 

12 

5 

42 

0.697 

White 

193 

85 

44 

Below mean on Grade 7 state mathematics 
assessment 15 

106 

41 

39 

0.398 

Above mean on Grade 7 state mathematics 
assessment 

104 

49 

47 

Below mean on Fall 2008 pretest (Promise 
Assessment) 15 

100 

32 

32 

0.015 

Above mean on Fall 2008 pretest (Promise 
Assessment) 

110 

58 
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AR is algebra ready. 

Note: Sample includes 35 schools and 21 1 AR online course participants. Tests of significance were conducted using a 
hierarchical generalized linear model that assumed a Bernoulli sampling distribution and logit link function. All models were 
adjusted for nesting of students within schools. 

a. The three groups of students were aggregated into a single group for analysis and presentation in order to protect student 
confidentiality. 

b. AR students enrolled in online course only. 

Source: Study team records based on online course activity archival data. Data for demographics and state mathematics 
achievement are from the Maine state department of education and supervisory unions in Vermont (2008). 

Mode of Instruction and Content of the Regular Mathematics Courses Offered 
to Students in Treatment and Control Schools 

The study team also collected data on the mathematics instruction received by students taking the 
regular grade 8 mathematics classes in treatment and control schools. In treatment schools, these 
were the N-AR students, and in control schools, these were both AR and N-AR students. The 
data sources included classroom materials, the teacher survey, class rosters, and site visits by 
members of the study team. 

Mode of Instruction for General Mathematics Courses in Treatment and Control Schools 

Site visits confirmed that the predominant mode of instruction in grade 8 mathematics 
classrooms in the treatment schools (that is, the courses provided to the N-AR students in 
treatment schools) was face-to-face delivery of content. None of the primary general 
mathematics curricula in treatment schools was computer based; computers were used only to 
supplement instruction in some classrooms. Analyses of classroom materials indicated that text- 
based programs were used in grade 8 classes in all 29 classrooms that provided data. The most 
commonly used program was Pearson-Prentice Hall Connected Mathematics (9 schools) 
followed by McDougal Littell Math Thematics (5 schools) and Glencoe Mathematics Course 3 
(5 schools). Other textbooks included Saxon Course 3 (. Algebra 1/2 ) and Foresman- Wesley 
Math. 
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The main mode of instruction in control schools (delivered to both AR and N-AR students) was 
also face-to-face, teacher-led instruction. The two most commonly used grade 8 mathematics 
programs in control schools were Connected Mathematics (9 schools) and McDougal Littell 
Math Thematics (8 schools). Other textbooks included Saxon Course 3 (. Algebra 1/2), Glencoe 
Mathematics Course 3, Glencoe MathScape, Pearson/Prentice Hall Algebra I, Holt Algebra I, 
and Foresman-Wesley Math. 43 

Content of General Mathematics Courses in Treatment and Control Schools 

Classroom materials were analyzed to compare the content of the general mathematics courses 
offered in treatment and control schools. Classroom materials included the name of the textbook 
used and any of the following that were available: course syllabi, curricular pacing guides, 
annotated tables of contents of mathematics textbooks, and course exams. Mathematics content 
experts on the study team coded the general grade 8 class materials, indicating the degree to 
which they focused on algebraic content (25%, 50%, 75%, and 100%). The coders used just four 
categories because detailed pacing information was available from too few schools to estimate 
more precisely (by, for example, the number of weeks spent on algebra). 

The course content delivered to N-AR students in the treatment schools included both general 
grade 8 mathematics content and algebra content. Among the treatment schools that provided 
classroom materials for the general grade 8 mathematics class (from which the online Algebra I 
course participants had been removed), 79% had a curricular focus on algebraic content of 50% 
or higher. Nearly one-third (31%) had a focus of 75% and 0% of treatment schools had a general 
grade 8 mathematics class with an algebraic content focus of 100% (as did the online Algebra I 
course offered to AR students in treatment schools). 

Among the control schools that provided grade 8 mathematics classroom materials, over 90% 
had a curricular focus on algebraic content of 50% or higher. More than one-third (35%) of the 
control schools had a focus on algebra of 75%, and 16% had an algebraic focus of 100%. Based 
on the review of state content standards and recruitment discussions with state and local 
educators, researchers expected that the general mathematics courses in control schools would 
include a substantial amount of algebraic content. However, without a separate comparison 
group of schools that was unaware of the study, there is no way to know whether the amount of 
algebra in the study control schools represents what is typical or whether the amount of algebra 
offered was affected by participation in the study. 

The target sample for the study was schools that did not offer a stand-alone Algebra I course to 
all of their AR students (schools that offered a separate Algebra I course to a limited number of 
AR students were considered eligible for the study). 44 The study team did not expect any of the 
control schools to offer a formal Algebra I course to all of their AR students. However, analysis 
of classroom materials indicated that in 7 control schools, most (94%) of AR students took a 
separate Algebra I course at their middle school or the local high school. In total, 45 AR students 

43 Further information about the number of schools that used each textbook and links to more information about 
each textbook are provided in appendix B, table B- 1 . 

44 Schools that provided Algebra I to grade 8 students in a limited way (by, for example, allowing students to take 
the course at the high school when scheduling worked out) were eligible for this study, because they did not offer 
the course in a way consistent with the goals of the study: to broaden access to Algebra I for all AR students in the 
school. 
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at control schools took Algebra, representing 20.3 percent of the total sample of AR students in 
control schools. 

Other Programmatic Features of the Intervention and Business as Usual 
Mathematics Instruction 

In addition to the mode of delivery and mathematics content, adoption of the online course in 
study schools introduced four potential differences from the business-as-usual grade 8 
mathematics programs: qualifications of teachers, staffing intensity, class size, and ability 
grouping of students. In the following sections, we describe the implementation of (1) the online 
course delivered to AR students in treatment schools, (2) the regular mathematics classes 
(delivered to N-AR students) in treatment schools, and (3) the regular mathematics classes 
delivered to both AR and N-AR students in control schools in terms of these four aspects. 

Teacher Qualifications 

As reported in Chapter 2 (see Table 2-12), a significantly larger proportion of online teachers 
(100%) were certified in secondary mathematics than the regular teachers in treatment schools 
(39%) and control schools (37%) (p = 0.003 and 0.003, respectively). In addition, significantly 
more online teachers held master’s degrees in mathematics education or mathematics than did 
regular teachers in treatment schools (38%) and control schools (20%) {p = 0.017 and 0.001, 
respectively). 45 The online teachers had taught Algebra I for an average of 9.5 years, 
significantly more than the average for regular teachers in treatment schools (2.3 years) and in 
control schools (3.7) (p = 0.001 and 0.008, respectively). The online teachers also had 
significantly more years of total teaching experience (22.9 years) than did the regular teachers in 
treatment schools (14.8 years) and in control schools (14.7) (p = 0.034 and 0.043, respectively). 

Staffing Intensity 

To capture staffing intensity, researchers used two measures: the number of adults associated 
with the students taking the online course and the overall grade 8 student to teacher ratio. 

All students taking the online course had access to both an online teacher and an in-class proctor; 
therefore, two adults were available to support each student enrolled in the online course. 
Because the proctor was also the regular grade 8 mathematics teacher in 80% of the treatment 
schools, both adults were qualified to provide mathematics instruction (albeit not necessarily 
Algebra I instruction in the case of all proctors). N-AR students in treatment schools had access 
to one mathematics teacher, yielding a staffing intensity of one teacher per N-AR grade 8 
student. Accounting for the online teacher and online course proctors who were not also grade 8 
mathematics teachers, the student-teacher ratio for all students (AR and N-AR) in treatment 
schools was 10.1 to 1 46 Students in control schools also had access to one mathematics teacher; 
the student to teacher ratio was 18.8 to 1. 


45 To maintain confidentiality of the participants, the number and percentage of online teachers with master’s 
degrees were suppressed for presentation purposes. 

46 The student-teacher ratio was calculated by dividing the total number of grade 8 students by the total number of 
adults per school involved in grade 8 mathematics. In treatment schools, the number of adults equals the sum of the 
number of grade 8 mathematics teachers, the online teacher, and online proctors who were not also grade 8 
mathematics teachers. In control schools, the number of adults is the number of grade 8 mathematics teachers. 
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Class Size 

For the online class, two measures of class size were used: the number of students per online 
section and the number of online students per school. The average number of students per online 
section was 24 (standard deviation = 6.0). 47 The average number of online students per school 
was 6.9; however, not all online students participated in the online course at the same time in the 
same physical space. The average number of online students who participated in the course at the 
same time and in the same space per treatment school was 3.9 (standard deviation = 1.9). 

The average class size of the general grade 8 mathematics classes in treatment schools (which 
did not include students taking the online Algebra I course) was 9.98. The average class size 
reported by grade 8 mathematics teachers in control schools was 1 1.69. The difference in the 
average class size of general grade 8 mathematics classes in treatment and control schools was 
not statistically significantly different (p = 0.259). 

Ability Grouping 

Nearly all of the students (97%) in treatment schools identified by their schools as algebra ready 
before the start of the study enrolled in the online course, and most of the online algebra course 
participants (87%) were AR students. Analyses of mathematics achievement scores confirmed 
that the students identified as AR were higher achieving than their N-AR peers who were not 
identified for the course (see chapter 2 and appendix A, figures A-2 and A-3). Therefore, it is 
safe to conclude that in treatment schools, AR and N-AR students were separated by ability 
level, as expected. 

However, AR and N-AR students were often not segregated into different physical classes in 
treatment schools. Site visits to treatment schools noted that online students sat in the general 
grade 8 mathematics classroom to take the online course in about 69% of schools. 

To capture the extent of ability grouping in control schools, researchers used class rosters and 
teacher survey reports to gauge the extent to which AR and N-AR students were assigned to the 
same teachers and were in the same sections. Of the 33 control schools, 18 (55%) had classes in 
which AR and N-AR students were integrated, 1 1 (33%) had classes in which students were 
separated by ability, and 4 (12%) had only AR students. 

Researchers surveyed teachers of the regular mathematics classes in treatment and control 
schools to understand their approaches to ability grouping and differentiating instruction. 
Although it appeared that students in the general grade 8 classrooms in treatment schools were 
on average lower in ability level and more homogenous than they would have been had the AR 
students been enrolled in their class, about 80% of teachers reported differentiating their 
instructional practices to a moderate or great extent. In control schools, about 75% of teachers in 
control schools reported differentiating their instructional practices to a moderate or great extent. 

When asked about giving students with higher ability accelerated material, such as Algebra I 
content, about 45% of the regular mathematics teachers in treatment schools reported doing so, 
compared with about 68% of the teachers in control schools. This difference was statistically 
significant (p = 0.024). 


47 This average includes both AR and non-AR students enrolled in the online course. 


56 



Summary of Key Implementation Findings 

The six programmatic features identified in chapter 1 were used to describe mathematics 
instruction in the treatment and control classrooms (table 3-5). 


Table 3-5. Programmatic Features of Intervention and “Business as Usual” Grade 8 
Mathematics Course in Study Schools 


Feature 

AR students in treatment 
schools 

N-AR students 
in treatment schools 

AR and N-AR students in 
control schools 

Mode of delivery 

Web based 

Face to face 

Face to face 

Course content 

Algebra I 

79% of schools’ curricula 
emphasized algebraic 
concepts 50% or more of the 
time 

Over 90% of schools’ curricula 
emphasized algebraic concepts 
50% or more of the time; 20% 
of AR students took separate 
Algebra I course 

Teacher 

100% certified to teach 

44% certified to teach 

43% certified to teach 

qualifications 

secondary math 

secondary math 

secondary math 

Staffing intensity 

2 staff per student (online 
and proctor); student to 
teacher ratio of 10.1 to 1 

1 teacher per student; student 
to teacher ratio of 10. lto 1 

1 teacher per student; student to 
teacher ratio of 18.8 to 1 

Class size 
(students per 
class) 

3.9 

10.0 

11.7 

Ability 

High-ability students 

Non-high ability students 

Students grouped together in 

grouping 

grouped together 

grouped together; 45% of 
teachers reported accelerating 
instruction for high-ability 
students 

58% of schools and separately 
in 33% of schools (12% of 
schools had only AR students); 
68% of teachers reported 
accelerating instruction for 
high-ability students 


AR is algebra ready. 


Analyses of archived data from the online course management system and data from the weekly 
proctor logs showed that the types and amount of communication between online course 
participants and their teachers and proctors did not match the initial expectations for the 
intervention. Online teachers did not communicate directly with all of their students about the 
course on a daily basis. The on-site proctor’s role was not intended to include mathematics 
instruction, but proctors reported that they spent approximately 50 minutes a week answering 
students’ questions about the mathematics content of the online course. 

The content of the online course was typical for an Algebra I course, and participating students 
varied in the amount of the course content they completed. The course completion rates indicate 
that 43% of AR students who participated took a “full” Algebra I course, and most (more than 
80%) completed more than half the course. 

The general mathematics classes in control and treatment schools also included considerable 
Algebra I content, especially for AR students in control schools. In the control schools, 20% of 
AR students took a full Algebra I course during the year, and AR students in over 90% of the 
control schools that provided course materials received mathematics instruction that had a 
curricular focus on algebra content of 50% or more. 
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In terms of teacher qualifications, 100% of online teachers were certified in secondary 
mathematics, compared with 41% of grade 8 control teachers and 39% of treatment teachers. The 
staffing intensity was higher in treatment schools than in control schools, because students in the 
online course had access to both their online teacher and a proctor (who was often the grade 8 
mathematics teacher). Students in face-to-face classes in both treatment and control schools had 
access to only the regular grade 8 mathematics teacher. Student to teacher ratios in treatment and 
control schools reflect these differences: the student to teacher ratio was 10.1 to 1 in treatment 
schools and 18.8 to 1 in control schools. Although an average of 6 AR students per treatment 
school were placed into the online course, the average class size of general grade 8 mathematics 
classes in treatment and control schools was not significantly different. 

With respect to ability grouping, removing the AR students from general grade 8 treatment 
classrooms and placing them in the online course created ability groupings (AR and N-AR 
students were enrolled in different courses). However, approximately 80% of the online course 
proctors were also the grade 8 mathematics teacher, and AR and N-AR students sat in the same 
physical classrooms in almost 70% of treatment schools. In 26 of the 31 control schools that 
provided data, AR and N-AR students were integrated in the same grade 8 mathematics classes. 
However, several fonns of within-class ability grouping occurred in these classrooms. In many 
cases, instruction was differentiated, with AR students receiving more instruction in Algebra I 
content than N-AR students. Teachers in control schools (who taught both AR and N-AR 
students in the same classes) were significantly more likely to report giving higher-ability 
students accelerated material than teachers in treatment schools (who taught N-AR students 
only). 

Descriptive analyses from multiple data sources indicate that the online course was implemented 
as intended in the 35 treatment schools. The course was the only mathematics course taken by 
the AR students during that academic year. Students were provided with courseware that was 
fully functional throughout the school year, the technology they needed to access the course from 
school (one-on-one access to a computer with a high-speed Internet connection), a class period 
for Algebra I online that met with the same frequency and duration as the regular grade 8 
mathematics course, and an on-site proctor who supervised AR students during their 
mathematics class period. However, the online teachers monitored student progress and 
interacted with students less than expected, and the proctors provided more instructional support 
to the online students than expected. This information provides a backdrop for the next chapter, 
which focuses on the impact of the intervention on student achievement and high school 
coursetaking in mathematics. 
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Chapter 4 Impact of Online Algebra 1 on Primary and Secondary 

Student Outcomes 

This chapter reports the results of the impact analyses conducted to address the study’s six 
research questions. It examines the impacts of online Algebra I on AR and N-AR students after 
one year of implementation. Student outcome data, including an algebra and general 
mathematics posttest and planned coursetaking information, were collected from all grade 8 
students in the participating schools in spring 2009. Follow-up coursetaking data were collected 
in spring 2010 for AR students only. 

The results presented in this chapter are based on an intent-to-treat analysis that includes all 
grade 8 students in the study schools at the time of baseline data collection (fall 2008), except 
those who were withdrawn from the study by their parents (nonconsenters) or their schools (“not 
testable” students). Thus, the impact estimates reflect the impact of assignment to treatment 
condition. 

The impacts were estimated using a multilevel regression model that uses all available 
observations from the treatment and control groups, including information on baseline 
covariates. The means reported for students in both treatment and control schools are the model- 
adjusted means, controlling for gender, free or reduced-price lunch status, special education 
status, prior state mathematics test scores, and school-level blocking variables (state and school 
size). 

For analyses of continuous variables (algebra and general mathematics posttest scores), impact 
estimates are presented in their original metric and as Hedges’ g effect sizes. For analyses of 
categorical variables (coursetaking), the impact of the intervention is the difference in the 
probability of participating in an advanced mathematics course sequence by students in treatment 
and control schools. For these analyses, the difference between treatment and control groups is 
presented as the difference in the average model-predicted probability of participating in an 
advanced mathematics course sequence for AR students in each condition, controlling for prior 
mathematics achievement; demographic characteristics (gender, free or reduced-price lunch 
status, and special education status); and school-level blocking variables (state and school size). 49 

For both types of analyses, the tables report the standard error and /t-valuc for each impact 
estimate. Some differences in group means or predicted probabilities could occur by chance. The 
standard error indicates the magnitude of the uncertainty about the true mean of each impact, 
given the number of schools and students in the analysis. The /;- value indicates the chance of 
obtaining an impact the size of the estimated impact if in fact there were no true impact. 


48 Hedges’ g calculations use a standard deviation that is pooled across the treatment and control groups (Hedges 
1981). Variances for the treatment and control groups used in the calculation of effect sizes were adjusted to account 
for the within and between imputation variance in the multiply-imputed datasets (Rubin 1987; Shafer & Graham 
2002 ). 

49 The predicted probabilities were calculated by first generating logit estimates at the treatment and control sample 
means (with all covariates grand mean centered) and then converting them into probabilities for the treatment and 
control groups separately. These group means and associated standard errors were generated in SAS using PROC 
GLIMMIX by requesting the LSMEANS output for the contrast for condition. 
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This chapter first presents the results of the analyses that tested the primary research questions, 
including the main impact results followed by a summary of primary findings and the sensitivity 
analyses that were conducted to test the robustness of the main results. It then presents the results 
of the analyses that tested the secondary research questions, including the main results and the 
results of sensitivity analyses. A fuller presentation of results for each impact model is provided 
in appendix D, including coefficients and standard errors for each predictor; corresponding odds 
ratios are also included for coursetaking outcomes. The results for all sensitivity analyses are 
presented in appendix E. 

Primary Impact Analyses: Impacts on Algebra Scores and High School 
Coursetaking by AR Students 

The results reported in this chapter are based on analysis of the AR student sample, where 
missing data were imputed using multiple imputation (see appendix F). All estimates are 
aggregates across all 10 multiply imputed datasets that account for variance between and within 
the imputed dataset. The analytic sample for the AR analyses includes 440 students enrolled in 
68 schools (218 students in 35 treatment schools and 222 students in 33 control schools). 

Algebra Scores at the End of Grade 8 

A two-level hierarchical model with students nested within schools was used to estimate the 
impact of having access to an online Algebra I course on AR students’ algebra assessment scores 
at the end of grade 8. To improve the precision of the impact estimates, researchers included 
students’ prior state mathematics test scores and background characteristics (gender, eligibility 
for free or reduced-price lunch, and special education status) as covariates in the model. School- 
level covariates included blocking variables (state and school size dummy variables). Except for 
the treatment status indicator, all covariates were centered on the grand mean. 

AR students in schools randomly assigned to offer the online Algebra I course scored higher on 
the algebra posttest than their counterparts in schools that did not receive the course (table 4-1). 
The average algebra score for AR students in treatment schools was 5.53 scale score points 
higher than the average score for AR students in control schools (effect size = 0.40). 


Table 4-1. Impact of Online Algebra I on Algebra Scores of AR Students in Treatment and 
Control Schools 


Mean in 

Mean in 



treatment schools 

control schools 

Estimated impact 


(standard deviation) 

(standard deviation) 

(standard error) p-value 

Effect size 

447.17 

441.64 

5.53* 0.001 

0.40 

(15.04) 

(12.29) 

(1.57) 



AR is algebra ready. 

* Two-tailed statistical significance. Because of a multiple comparison adjustment that accounts for two primary analyses, a p- 
value less than 0.025 is considered statistically significant. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). The treatment 
group and control group means are the model-adjusted mean scores for AR students, controlling for all covariates in the impact 
model. The effect size was calculated using a pooled standard deviation of the outcome for AR students in treatment and 
control schools that incorporates both within and between imputation variance ( SD = 13.78). 

Source : Algebra scores on study-administered Promise Assessment posttest. 
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High School Coursetaking 

A two-level hierarchical model that is appropriate for binary outcomes was used to estimate the 
effect of having access to online Algebra I in grade 8 on the likelihood of participating in an 
advanced mathematics course sequence in high school. The model used is a hierarchical 
generalized linear model that assumes a Bernoulli sampling distribution and logit link function 
(Raudenbush and Bryk 2002; McCullagh and Nelder 1989). 

The outcome measure for this analysis is participation in an advanced course sequence, based on 
the grade 9 mathematics courses taken, grades earned, and the course planned for grade 10. 
Students were considered advanced if they took a course above Algebra I in grade 9, passed their 
grade 9 course with a grade of C or higher, 50 and enrolled in Algebra II or higher for grade 10. 

The impact of the online course on the likelihood of participating in an advanced course 
sequence was assessed with a hierarchical generalized linear model analysis that, like the model 
used to estimate the impact on algebra scores, included student- and school-level covariates. At 
the student level, the model controlled for students’ prior state mathematics assessment scores 
and demographic characteristics (gender, eligibility for free or reduced-price lunch, and special 
education status). At the school level, the model included the blocking variables (state and school 
size dummy variables) and the treatment status indicator. All covariates in the model (except the 
treatment status indicator) were centered on the grand mean. 

The results indicate that AR students from schools randomly assigned to offer the online Algebra 
I course were significantly more likely to follow an advanced mathematics course sequence than 
their AR counterparts in schools that did not offer the course (table 4-2). Specifically, the 
average probability of participating in an advanced course sequence was 0.26 for AR students 
from control schools and 0.5 1 for AR students from treatment schools. The intervention yielded 
a difference in the probability of participating in an advanced course sequence of 0.25, meaning 
that AR students from treatment schools were nearly twice as likely to participate in an advanced 
mathematics course sequence as AR students in control schools. 

Table 4-2. Predicted Probability of AR Students Participating in an Advanced 
Mathematics Course Sequence in High School 


Treatment school 
(standard error) 

Control school 
(standard error) 

Difference in probability 
attributed to intervention 

p-value 

0.51 

0.26 

0.25* 

0.007 

(0.07) 

(0.05) 


* Two-tailed statistical significance. Because of a multiple comparison adjustment that accounts for two primary analyses, a p- 
value less than 0.025 is considered statistically significant. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 students (218 treatment, 222 control). Coursetaking 
patterns were coded as representing successful completion of a course above Algebra I in grade 9 and enrollment in Algebra II 
or a higher course in grade 10) or not. The probabilities are the average model-predicted probabilities, controlling for all 
covariates specified for the model. 

Source: Coursetaking data collected from high schools AR study students attended in 2009/10. 


50 If students took more than one mathematics course in grade 9, they had to have earned a grade of C or 
better on the more advanced grade 9 course to meet this criterion. 
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Results of Sensitivity Analyses 

A set of sensitivity analyses was conducted to test the robustness of these results (see appendix 
E, tables E-l and E-8). For the analysis of algebra scores at the end of grade 8, the direction of 
the treatment effect did not change under several different models and assumptions. As noted in 
Chapter 2, there was a difference by condition on the study-administered pretest such that AR 
students in treatment schools had higher scores and spent more time on the pretest than AR 
students in control schools. As a result, sensitivity model E.2 demonstrates that the magnitude of 
the effect decreased when the post-random assignment study-administered pretest was used as 
the baseline measure of prior achievement instead of the state mathematics assessment scores 
(see appendix E, model E.2). To examine whether the treatment effect on algebra posttest scores 
could be partially or completely explained by time spent on the posttest, sensitivity model E.7 
included posttest testing time as a covariate. Results confirmed that the effect of the intervention 
on AR students’ algebra posttest scores was not explained by testing time (see appendix E, 
model E.7). 

For the analysis of coursetaking, sensitivity models that controlled for the study-administered 
pretest as a baseline achievement covariate instead of the state mathematics scores but were 
otherwise identical yielded results different from those reported in the benchmark analyses. 

Using the benchmark model but controlling for the pretest instead of the state mathematics 
scores indicated that the effect of the intervention was not statistically significant (p = 0.037), 
using an alpha value of 0.025 that accounts for multiple comparisons (see appendix E, model 
E.9). The results of all other sensitivity analyses conducted for high school coursetaking were 
consistent with the main results reported in this chapter. 

Secondary Impact Analyses 

Impacts on General Mathematics Outcomes for AR Students 

The primary focus of the impact of access to online Algebra I for AR students is on their algebra 
achievement at the end of grade 8 and subsequent high school coursetaking. A secondary 
outcome is their achievement at the end of grade 8 on a general mathematics test. A significant 
negative effect on this outcome could signal a potential down side of offering an online Algebra I 
course to AR students. 

To test whether access to online Algebra I affected AR students’ general mathematics scores, 
researchers used the same two-level model used to estimate the impact of online Algebra I on 
AR students’ algebra scores. The results revealed no significant difference by condition (effect 
size = 0.14) (table 4-3). 51 


51 The lack of a significant difference does not definitively show that general mathematics scores for AR 
students in treatment and control schools were equivalent. It simply implies that the difference was not 
large enough to be distinguished from chance, given the size of the sample. 
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Table 4-3. Impact of Online Algebra I on AR Students’ General Mathematics Scores at the 
End of Grade 8 


Treatment schools 
(standard deviation) 

Control schools 
(standard deviation) 

Estimated impact 
(standard error) 

p-value 

Effect size 

361.42 

357.82 

3.60 

0.204 

0.14 

(24.79) 

(25.43) 

(2.80) 




AR is algebra ready. 


Note: Sample includes 68 schools (35 treatment, 33 control) and 440 students (218 treatment, 222 control). The treatment and 
control group means are the model-adjusted mean scores for AR students, controlling for all covariates in the impact model. 
Result is not statistically significant. The effect size was calculated using a pooled standard deviation of the outcome for AR 
students in treatment and control schools that incorporates both within and between imputation variance ( SD = 25.22). 

Source : General mathematics scores on study-administered Promise Assessment posttest. 


Impacts on N-AR Students 

Three secondary analyses were conducted to determine the impact of offering Algebra I online to 
AR students on N-AR students’ outcomes. The three outcomes of interest were algebra and 
general mathematics scores at the end of grade 8 and planned grade 9 courses. For these 
analyses, N-AR students in treatment schools were compared with N-AR students in control 
schools. 

Multiple imputation was used to create 10 datasets with missing data imputed (see appendix F). 
All estimates are aggregates across all 10 datasets that account for the variance between and 
within the imputed dataset. The analytic sample for the N-AR analyses included 1 ,445 students 
enrolled in 68 schools (744 in treatment schools and 701 in control schools; 4 control schools 
had no N-AR students). 

Impact on Test Scores. A two-level hierarchical model with students nested within schools was 
used to estimate the impacts of online Algebra I on N-AR students’ algebra and general 
mathematics assessment scores at the end of grade 8. The models used were identical to those 
described earlier for AR students for the same achievement outcomes. 

The results of this analysis indicate no significant differences in algebra or general mathematics 
posttest scores between N-AR students in schools randomly assigned to receive the online 
Algebra I course and their N-AR counterparts in control schools (table 4-4). The impact of 
online Algebra I translates to an effect size of 0.06 on algebra scores and 0.02 on general 
mathematics scores, neither of which is statistically significant. 
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Table 4-4. Impact of Online Algebra I on N-AR Students’ Algebra and General 
Mathematics Scores at the End of Grade 8 


Subject area 

Treatment schools 
(standard 
deviation) 

Control schools 
(standard 
deviation) 

Estimated impact 
(standard error) 

p-value 

Effect 

size 

Algebra 

430.76 

(15.36) 

429.80 

(15.64) 

0.96 

(1.25) 

0.443 

0.06 

General Mathematics 

324.86 

(28.42) 

324.21 

(30.04) 

0.65 

(2.41) 

0.789 

0.02 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The treatment and control group 
means are the model-adjusted mean scores for AR students, controlling for all covariates in the impact model. Results are not 
statistically significant. Effect sizes were calculated using a pooled standard deviation of the outcome for N-AR students in 
treatment and control that incorporates both within and between imputation variance ( SD = 15.50 for Algebra and 29.39 for 
General Mathematics). 

Source: Algebra and general mathematics scores on study-administered Promise Assessment posttests. 


Impact on Planned Grade 9 Courses. Planned grade 9 courses for N-AR students were coded 
as 1 for intermediate (a course at or above Algebra I) and 0 for not intermediate (a course below 
Algebra I, such as Pre-algebra). To estimate the effect of offering online Algebra I (in grade 8) to 
AR students on N-AR students’ probability of enrolling in Algebra I in grade 9 (as per an 
intermediate course sequence), researchers used a two-level hierarchical model for binary 
outcomes. The differences between students in treatment and control school were not statistically 
significant (table 4-5). 

Table 4-5. Predicted Probability of N-AR Students Enrolling in Intermediate Mathematics 
Course Sequence in Grade 9 


Treatment schools 

Control schools 

Difference in probability 


(standard error) 

(standard error) 

attributed to intervention 

p-value 

0.89 

(0.04) 

0.79 

(0.06) 

0.10 

0.099 

N-AR is not algebra ready. 


Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. 

Probabilities are the average model-predicted probabilities, controlling for all covariates specified for the model. Result is not 
statistically significant. 

Source: Planned courses indicated by study students at end of Grade 8. 


Results of Sensitivity Analyses 

A set of sensitivity analyses was conducted to test the robustness of these results (see appendix 
E, tables E-22 and E-27). With one exception, the effect of online Algebra I did not change under 
different models and assumptions. The analyses did not detect significant effects on AR students’ 
general mathematics scores, on N-AR students’ algebra or general mathematics scores, or the 
likelihood of enrolling in an intermediate course sequence in high school. 

The one exception was for N-AR students’ planned grade 9 coursetaking. One sensitivity 
analysis (model E.23, which uses complete case analysis instead of multiple imputation for 
missing data) revealed a statistically significant difference in favor of students in treatment 
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schools. Specifically, N-AR students from treatment schools were significantly more likely to 
enroll in an intermediate level grade 9 course than N-AR students from control schools {p = 
0.035; see table E-27). The pattern was similar in the main analysis and the other sensitivity 
analyses, although the difference reached the level of statistical significance only in this analysis. 
Nevertheless, all of the sensitivity analyses (including this one) confirm that there were no 
significant negative effects of the intervention on AR students’ general mathematics scores or on 
any of the N-AR students’ outcomes. 
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Chapter 5 Exploratory Analyses of AR Students' High School 

Coursetaking 

The main results in chapter 4 indicated that AR students in schools that offered an online 
Algebra I course to broaden access to Algebra I in grade 8 were significantly more likely to 
follow an advanced mathematics course sequence in high school than AR students in control 
schools. This chapter examines two exploratory questions intended to further understand the 
main results regarding AR students’ coursetaking. It draws on data on AR students’ planned 
grade 9 mathematics classes and their actual grade 9 and grade 10 mathematics course 
sequences. 

Exploratory Research Questions 

Exploratory Research Question 1: How does access to online Algebra I in grade 8 affect the 
likelihood that AR students sign up for advanced courses in grade 9? This exploratory analysis 
examines the impact of access to online Algebra I in grade 8 on students’ planned grade 9 
mathematics course enrollments, using data provided by schools at the end of grade 8. The 
outcome of interest is whether AR students in treatment schools are more likely than their 
counterparts in control schools to enroll in a course for grade 9 that if successfully completed, 
would place them on an advanced coursetaking trajectory. 

Exploratory Research Question 2: How does access to online Algebra I in grade 8 affect the 
likelihood that AR students double up, or take more than one mathematics course a year, in 
grade 9 or 10? If access to online Algebra I in grade 8 enables students to enter high school 
farther along in the mathematics course progression, the intervention may decrease the 
probability that students double up on courses in grade 9 or 10. In addition to analyzing this 
question, researchers conducted a descriptive analysis of the coursetaking patterns that involve 
doubling up on mathematics courses. This analysis examined the different pathways by which 
students in treatment and control schools get on pace to complete Algebra II by the end of grade 
10 . 

Results 

Exploratory Research Question 1 

To estimate the effect of having access to online Algebra I in grade 8 on the probability of 
enrolling in a course above Algebra I in grade 9, researchers employed the same two-level 
hierarchical generalized linear model used in the main analysis of high school coursetaking (see 
chapter 4). 

The results of this analysis show that AR students in schools that offered the online Algebra I 
course were significantly more likely to register for an advanced grade 9 mathematics course 
than their AR counterparts in schools that did not offer the course (table 5-1). The average 
predicted probability of enrollment in an advanced mathematics course by AR students was 0.16 
in control schools and 0.54 in treatment schools. The intervention thus yielded a difference in the 
probability of enrolling in a grade 9 course above Algebra I of 0.38, meaning that AR students in 
treatment schools were 3.38 times more likely than AR students in control schools to enroll in an 
advanced grade 9 course. 
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Table 5-1. Predicted Probability of AR Students Enrolling in Advanced Mathematics 
Course in Grade 9 


Treatment schools 

Control schools 

Difference in 
probability attributed 


(standard error) 

(standard error) 

to intervention 

/7-value 

0.54 

0.16 

0.38* 

0.005 

(0.11) 

(0.07) 




AR is algebra ready. 

* No adjustments for multiple comparisons were applied for the secondary impact analyses, thus a /(-value less than 0.05 is 
considered statistically significant. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Probabilities are the average model-predicted probabilities, controlling for all 
covariates specified for the model. 

Source: Planned courses indicated by study students at end of Grade 8. 


Exploratory Research Question 2 

If access to online Algebra I in grade 8 enables students to enter high school farther along in the 
mathematics course progression, the intervention may affect the probability that students double 
up on full-year mathematics courses in grade 9 or 10. To address this question, researchers tested 
the impact of access to online Algebra I in grade 8 on the likelihood of taking two full-year 
mathematics courses in either grade 9 or grade 10 (that is, doubling up on mathematics 
coursework). The outcome for this analysis was a binary indicator of doubling up, where 
students were assigned a 1 if they doubled up in grade 9 or planned to do so in grade 10 and a 0 
if they did neither. A total of 96 students — 21% of the AR student sample across conditions — 
were coded as doubling up. 

The analytic model testing whether the intervention impacted whether AR students doubled up in 
grade 9 or 10 was a hierarchical generalized linear model that accounted for the nesting of 
students within schools and included the same student- and school-level covariates as the impact 
models, with the exception of special education status. The results indicate that AR students 
from treatment schools were significantly less likely to double up on mathematics courses than 
AR students from control schools (table 5-2). Access to online Algebra I thus appeared to 
decrease the likelihood that students take more than one full-year mathematics course in grade 9, 
grade 10, or both grades. 


52 Special education status was removed from the analytic model because no students who were deemed eligible for 
special education services doubled up in mathematics courses in either grade 9 or 10, causing mathematical errors in 
the estimation of the treatment effect. 
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Table 5-2. Predicted Probability of AR Students Doubling Up on Full-Year Mathematics 
Courses in Grade 9 or 10 


Treatment schools 

Control schools 

Difference in 
probability attributed 


(standard error) 

(standard error) 

to intervention 

/7-value 

0.10 

0.23 

-0.13* 

0.033 

(0.03) 

(0.06) 




AR is algebra ready. 

* No adjustments for multiple comparisons were applied for the secondary impact analyses, thus a /(-value of less than 0.05 is 
considered statistically significant. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Probabilities are the average model-predicted probabilities, controlling for all 
covariates specified for the model. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Taken together with the results of the main analysis presented in chapter 4, this exploratory 
analysis indicates that in addition to affecting whether students appear positioned to complete an 
advanced course sequence by the end of high school at the end of grade 9, offering Algebra I as 
an online course in grade 8 may also affect how they do so. Students with access to online 
Algebra I in grade 8 were less likely to double up on mathematics courses in grades 9 or 10 than 
their counterparts in control schools who, by and large, did not have access to Algebra I in grade 
8 . 
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Chapter 6 Summary of Key Findings 

This chapter summarizes the key findings, identifies the study limitations, and suggests 
directions for future research. 

Key Findings 

Primary Questions and Results 

The goal of this study was to measure the effects of broadening students’ access to Algebra I by 
offering of an online course in schools that do not typically offer the course. The primary 
analyses assessed the impact of offering an online Algebra I course on AR students’ algebra 
achievement at the end of grade 8 and their likelihood of participating in an advanced 
mathematics course sequence in high school. The results indicate that the intervention had a 
positive effect on both outcomes. On algebra achievement at the end of grade 8, AR students 
outperformed their control group peers by approximately 5.5 scale score points (effect size = 
0.40, p = 0.001). With regard to high school coursetaking, AR students who had access to online 
Algebra I in grade 8 were about twice (1.96 times) as likely to participate in an advanced high 
school course sequence than their peers in schools where Algebra I is typically not offered to 
grade 8 students (p = 0.007). 

Secondary Questions and Results 

The results of the secondary impact analyses suggest that there were no significant negative (or 
positive) effects on AR students’ general mathematics achievement at the end of grade 8. Having 
access to Algebra I in grade 8 through an online course (instead of taking the general grade 8 
mathematics course) did not reduce students’ scores on a general mathematics assessment. This 
finding has policy relevance, because middle schools participate in state accountability 
assessments that test a range of general mathematics areas that may include but are not limited to 
algebraic concepts. 

Secondary impact analyses of outcomes for N-AR students suggest that the implementation of 
online Algebra I in the study schools did not have any discernible effects on students who were 
not considered eligible for the course. Specifically, there were no treatment effects on N-AR 
students’ algebra or general mathematics achievement at the end of grade 8. Moreover, the 
probability of a N-AR student enrolling in a grade 9 mathematics course at or above Algebra I — 
an indicator of participation in an intermediate course sequence in high school — was not 
significantly different for students from treatment and control schools. Taken together, the 
analyses of N-AR students suggest that although schools’ grade 8 mathematics programs were 
altered with the adoption of the online Algebra I course, students who remained in the general 
grade 8 mathematics class did not learn less algebra or have a lower chance of enrolling in 
Algebra I as grade 9 students than they would have had their school not adopted the online 
course for their AR students. 

Conclusions Based on Primary and Secondary Findings 

This combination of findings is consistent with the framework presented in Chapter 1 for 
evaluating the effectiveness of using an online course to broaden students’ access to Algebra I. A 
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successful intervention in this context was defined as one that yielded positive impacts on either 
end-of-grade 8 algebra achievement or subsequent high school coursetaking for AR students, 
with no significant negative side effects on AR students’ general mathematics scores or on any 
achievement or coursetaking outcomes for N-AR students. The results showed that AR students 
with access to online Algebra I in grade 8 outperformed their counterparts in control schools on 
an end-of-year algebra assessment and were more likely to follow an advanced course sequence 
in high school. There were no side effects of the course on AR students’ end-of-year general 
mathematics achievement or on any of the N-AR students’ measured outcomes. Thus, the results 
suggest that offering an online course to AR students in grade 8 is an effective way to broaden 
access to the specific course, and later, to more challenging mathematics course opportunities, 
for students in schools that do not typically offer Algebra I to eighth graders. 

The study was designed to provide information to educators who are looking for ways to offer a 
key gateway course (Algebra I) to their grade 8 students who are ready for it, but for various 
reasons cannot typically offer full access to the course in a standard or traditional way. The goal 
for the intervention was to not only have an impact on AR students’ short-term algebra 
knowledge, but to also influence a sequence of mathematics opportunities and outcomes over 
time. The hypothesis associated with the primary questions for this study was that offering an 
online Algebra I course would benefit student outcomes in contrast to the mathematics 
instruction they would have received in absence of the online course. It may seem obvious that 
students with access to an online Algebra I course in grade 8 should leam more algebra and take 
more advanced courses earlier in high school than those that do not. For multiple reasons, 
however, the results observed in the primary and secondary analyses were not necessarily 
obvious and addressed gaps in the research base. 

First, before this study, there was no prior rigorous evidence that an online version of a formal 
Algebra I course could be offered to grade 8 students by schools that do not typically offer the 
course, in terms of technology and content support. Second, though the logistical implementation 
of the course went as planned, just under half (43%) of the AR students who enrolled in the 
course fully completed it, meaning that many of the AR students in the treatment group were not 
exposed to the entire course. At the same time, AR students in control schools were exposed to a 
substantial amount of algebraic content in the context of their general mathematics classes, and 
one out of five AR control students actually did take a formal Algebra I course. Despite these 
circumstances, this study still demonstrated that the intervention as implemented is more 
effective in promoting students’ success in mathematics than existing practices in these schools. 

Exploratory Analyses and Results 

Two sets of exploratory analyses examined AR students’ high school coursetaking more closely. 
First, the effect of access to online Algebra I in grade 8 on the likelihood of initial enrollment in 
an advanced course in grade 9 was examined using planned course enrollments. Like the 
analyses of coursetaking after a full year of high school, this exploratory analysis showed that 
AR students in treatment schools were more than three times as likely to enroll in an advanced 
course (that is, a course above Algebra I) than AR students in control schools (predicted 
probabilities = 0.54 and 0.16 for AR students in treatment and control schools, respectively; p = 
0.005). 
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A second exploratory analysis tested the effect of online Algebra I in grade 8 on the likelihood 
that AR students doubled up on mathematics courses in grade 9 or 10. According to the data 
collected, at least some high-achieving students without access to Algebra I in grade 8 get 
themselves on track for an advanced course sequence by doubling up on their mathematics 
courses. Twenty-one percent of the AR sample (a total of 96 students) took more than one full- 
year mathematics course in grade 9, planned to do so in grade 10, or both. The results indicate 
that access to online Algebra I in grade 8 significantly decreased the likelihood that students 
doubled up. Specifically, the results indicate that students in control schools (predicted 
probability = 0.23) were more than twice (2.3 times) as likely to double up in mathematics 
courses in grade 9 or grade 10 than AR students in treatment schools (predicted probability = 
0.10; p = 0.033). If the students who planned to double up did successfully complete Algebra II, 
by the end of grade 10 (beyond the duration of the study), they could feasibly “catch up” to their 
peers categorized as advanced coursetakers for the purpose of the study’s main analysis at the 
beginning of grade 1 1 . From an instructional and developmental perspective, however, doubling 
up on high school mathematics courses may be a poorer option than sequential engagement in 
the course content. A longer study is needed to determine whether beginning this course 
sequence in grade 8 impacts the quality of students’ mathematics learning throughout the course 
sequence and whether the various pathways do in fact continue (or converge) throughout high 
school and beyond. 

In sum, the exploratory analysis findings supplement the main analyses and suggest that in 
addition to affecting whether students pursue an advanced course sequence, the intervention had 
an impact on how students enter an advanced course pathway — that is, access to Algebra I 
through an online course in grade 8 can reduce the need to double up and thus may allow high- 
achieving students who would otherwise double up to take other high school courses and focus 
on one mathematics content area at a time 

Limitations of the Study and Future Research Directions 

This study was conducted with a sample of schools in Maine and Vermont that met the eligibility 
criteria for participation and agreed to take part in a random assignment study. Many of these 
schools were small (48% had grade 8 enrollments of less than 17 students), and 90% were in 
rural areas. Analyses of ECLS-K data (U.S. Department of Education 2009a) indicated that a 
significant proportion of schools do not offer Algebra I to grade 8 students (approximately 16% 
nationally); moreover, the proportion of schools in rural areas with limited access to Algebra I is 
higher than in urban and suburban areas. Still, it is not clear whether the study schools represent 
small rural schools located in other parts of the region or country or the extent to which the 
results observed in these schools generalize to other schools interested in using online courses to 
expand access to Algebra I to grade 8 students. 

Although the consent rates and response rates were high (greater than 95% in the AR sample and 
greater than 85% in the N-AR sample), they were not 100%. Multiple imputation was used to 
adjust for any bias nonresponse might introduce, but it is not impossible that bias was 
nonetheless present. 

One potential source of bias in the study was the difference by condition observed on the study- 
administered pretest, on which AR students from treatment schools both scored higher and spent 
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more time than did AR students from control schools. However, the availability and use of 
mathematics achievement scores from the state assessments taken the year prior to the study 
mitigated the potential threat that these differences might otherwise cause the study findings. 

The online course chosen, Class. corn’s Algebra I course, is similar in content and focus to the 
offerings of other providers. However, it is not clear that similar results would have been 
observed had another course provider been chosen. Moreover, the results observed in this study 
cannot necessarily be generalized to more recently developed online courses. 

For all these reasons, replication of this study is necessary to gain a better understanding of the 
potential impacts of using an online course to expand access to Algebra I to grade 8 students. In 
particular, future studies should examine longer-tenn effects of access to online Algebra I in 
grade 8 — through high school, college, and even beyond. This study included a one-year follow- 
up to track students from grade 8 into high school. A longer study is needed to assess whether 
access to online Algebra I in grade 8 continues to have an impact on participation in advanced 
mathematics coursetaking through the end of high school. 

As the use of online courses continues to increase in U.S. schools, future research should 
continue to study their effects on student coursetaking patterns and achievement in key content 
areas. Further investigation of the effectiveness of online courses should contrast the offering of 
them with various relevant business-as-usual situations. These include school settings where 
students’ lack of access to specific courses (where the control group does not take the course) as 
well as school settings where particular courses are oversubscribed or taught by under-qualified 
or uncertified teachers (where the control group would take a standard face-to-face version of the 
online course). 

Schools around the country, particularly those in rural areas, are in search of innovative ways to 
expand their course offerings. To address this need, this study focused on the use of an online 
course to provide access to Algebra I in schools that do not typically offer the course in grade 8. 

It did not compare the effects of taking online Algebra I versus a standard face-to-face version of 
the course in grade 8, and the results should not be interpreted to indicate that offering online 
Algebra I is better than (or as good as) offering a face-to-face Algebra I course to eighth graders. 
In addition, given that the study compared the offering of an online Algebra I course to a lack of 
(or limited) access to Algebra I in grade 8, it is not possible to isolate the portion of the observed 
effects that is due to the fact that the course was online. As noted in the earlier description of the 
intervention under test, the content of the course (Algebra I) cannot be untangled from the mode 
of instruction (online). Thus it is possible that broadening access to any type of formal Algebra I 
course to AR grade 8 students would yield similar effects. 

This study is the first of its kind to rigorously evaluate the impact of offering an online version of 
Algebra I in schools that otherwise do not typically offer the course, even though they have 
students who are ready to take it. For educators and students facing similar challenges, the results 
of this study may be particularly infonnative and promising. 
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APPENDIX A: STUDY DESIGN, STUDY SAMPLES, AND 
STATISTICAL PRECISION 


This appendix provides details about the design and implementation of the study, including 
in formation on the school and student samples and estimates of the study’s statistical precision 
based on the data used in the analysis. 

School and Student Samples 

The CONSORT flow chart (figure A-l) identifies the number of students on whom pretest and 
posttest data were collected and the number of students included in the analytic samples for the 
analysis of algebra posttest scores, coursetaking outcomes, and general mathematics scores. 

Two types of students were defined as out of scope of the sample. The parents of 68 students (5 
algebra ready and 63 non-algebra ready) withdrew them from data collection. Forty-six N-AR 
students were withdrawn from the study by their schools because of their special education 
status. These students were deemed “not testable” and out of scope of the sample. 

Among students who were in scope of the sample, there were three main reasons for a missing 
score on the pretest or the algebra and general mathematics posttests: 

• Persistent absence on testing days 

• A score based on low testing time, defined as less than five minutes, which was 
considered invalid by the test developer 

• A move out of the participating school (and not into another participating school) during 
the 2008/09 school year. 5 ’ 

Information on planned grade 9 courses was missing primarily because of lack of availability at 
the time of collection (for example, the course registration was not yet set at the time of 
collection). Information on students who had moved out of a participating school was also 
generally not available. 

Infonnation on actual high school courses taken was missing because schools did not provide the 
data when requested or individual student data were not available from the high schools because 
the student had moved. Data were provided by 63 of 66 high schools (95%) within the time 
frame (June- July 2010). 


53 Students who moved out of a participating school and into another participating school were retained in the 
sample and remained in their original school for the purpose of clustering and analysis. 
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Figure A-l. CONSORT Flow Chart and Sample Tracking 

Enrollment 


Assessed for eligibility 

School A= 264 

„ Excluded (not eligible) or refused 
School N= 193 


Recruited 
School N= 71 


Excluded 


School N= 3 


(dropped out or ruled ineligible) 



1 


Randomized and eligible schools and 
students 
School A =68 
AR student N =468 
N-AR student N = 1,554 


Allocation 

ALLOCATED TO TREATMENT 

ALLOCATED TO CONTROL 

School n = 35 

School n = 33 a 

AR student n = 230 

AR student n = 238 

Nonconsent n = 0 (0%) 

Nonconsent n = 5 (2.1%) 

Moved during summer 2008 n = 12 (5.2%) 

Moved during summer 2008 n= 11 (4.6%) 

AR student (fall 2008) total n = 218 

AR student (fall 2008) total n = 222 

N-AR student n = 782 

N-AR student n = 772 

Nonconsent n = 29 (3.7%) 

Nonconsent n = 34 (4.4%) 

Not testable n = 9 (l.l%) b 

Not testable n = 37 (4.7%) b 

N-AR student (fall 2008) total n = 744 

N-AR student (fall 2008) total n = 701 

Baseline 

Baseline mathematics achievement (grade 7 state assessment scores ) 

School n = 35 

School n = 33 

AR students (« = 218) 

AR students (; n = 222) 

Non-missing: n > 215 (>99%) 

Non-missing: n = 219 (99%) 

Missing: n< 3 (<1%) 

Missing: n = 3 (1%) 

N-AR students (n = 744) 

N-AR students (n = 701) 

Non-missing: « = 721 (97%) 

Non-missing: n = 682 (97%) 

Missing: n = 23 (3%) 

Missing: n = 19 (3%) 
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Pretest — general mathematics 

School n = 35 


School n = 33 

AR students (« = 218) 


AR students (« = 222) 

Not tested: n< 3 


Tested: n = 218 



Not tested: n = 4 



Absent n = 4 

Response rate = >99% (>215/218) 


Response rate = 98% (218/222) 

Missing rate = <1% (<3/21 8) 


Missing rate = 2% (4/222) 

N-AR students ( n = 744) 


N-AR students (n = 701) 

Tested: n = 724 (of these, 10 cut because students spent 

Tested: n = 676 (of these, 6 cut because students spent 

less than five minutes on test) 


less than five minutes on test) 

Not tested: n = 20 -> 


Not tested: n = 25 

Absent n = 20 


Absent n = 25 

Response rate = 97% (724/744) 


Response rate = 96% (676/701) 

Missing rate = 4% (30/744) c 


Missing rate = 4% (31/701)° 

Follow-up 

Posttest — algebra 

School n = 35 

School n = 33 

AR students (; n = 218) 

AR students (n = 222) 

Not tested: n < 3 

Not tested: n < 3 

Response rate = >99% (>215/218) 

Response rate = >99% (>219/222) 

Missing rate = <1% (<3/21 8) 

Missing rate = 2% (4/222) 

N-AR students (n = 744) 

N-AR students (n = 701) 

Tested: n = 703 (of these, 73 cut because student 

Tested: n = 656 (of these, 45 cut because student spent less 

spent less than five minutes on the test) 

than five minutes on the test) 

Not tested: n = 41 

Not tested: n = 45 

Absent n — 13 

Absent n = l 

Moved n = 28 

Moved n = 38 

Response rate = 94% (703/744) 

Response rate = 94% (656/701) 

Missing rate = 15% (1 14/744) c 

Missing rate = 13% (90/701)° 

Posttest — general mathematics 

School n = 35 

School n = 33 

AR students (n = 218) 

AR students (n = 222) 

Not tested: n < 3 

Tested: n = 219 


Not tested: n = 3 


Absent n = 0 


Moved n = 3 

Response rate = >99% (>215/218) 

Response rate = 99% (219/222) 

Missing rate = <1% (<3/21 8) 

Missing rate = 1% (3/222) 

N-AR students (n = 744) 

N-AR students (n = 701) 

Tested: n = 703 (of these, 13 cut because student 

Tested: n = 656 (of these, 4 cut because student spent less 

spent less than five minutes on the test) 

than five minutes on the test) 

Not tested: n = 41 

Not tested: n = 45 

Absent n= 13 

Absent n — 1 

Moved n = 28 

Moved n = 38 

Response rate = 95% (703/744) 

Response rate = 94% (656/701) 

Missing rate = 7% (54/744) c 

Missing rate = 7% (49/701)° 


A3 





Planned grade 9 mathematics courses 

School n = 35 
AR students (/? = 218) 
Not collected: n = 3 

School n = 33 
AR students (n = 222) 

Collected: n = 21 1 

Not collected: n= 11 (not known at time of collection or 
student moved) 

Response rate = 99% (215/218) 
Missing rate =1% (3/218) 

Response rate = 95% (211/222) 
Missing rate = 5% (11/222) 

N-AR students ( n = 744) 
Collected: n = 697 
Not collected: n = 47 
Not available /? = 19 
Moved n = 28 

N-AR students (n = 701) 
Collected: n = 652 
Not collected: n = 49 
Not available n = 1 1 
Moved n = 38 

Response rate = 94% (697/744) 
Missing rate = 6% (47/744) 

Response rate = 93% (652/701) 
Missing rate = 7% (49/701) 

High school coursetaking (actual grade 9 courses and grades, planned grade 10 courses) 

AR students (/? = 218) 

Collected: n =211 

Not collected: n = l (not provided or student moved) 

AR students (n = 222) 

Collected: n = 216 

Not collected: n = 6 (nonconsent, not provided, or student 
moved) 

Response rate = 97% (211/218) 
Missing rate = 3% (7/218) 

Response rate = 97% (216/222) 
Missing rate = 3% (6/222) 

Analysis 

Algebra posttest 

AR analysis sample 

School n = 35 
Student n = 218 

Fewer than 4 posttest scores imputed 

AR analysis sample 

School n = 33 
Student n = 222 
4 posttest scores imputed 

N-AR analysis sample 

School n = 35 

Student n = 744 

1 14 posttest scores imputed 

N-AR analysis sample 

School n = 29 a 
Student n = 701 
90 posttest scores imputed 

General mathematics posttest 

AR analysis sample 

School n = 35 
Student n = 218 

Fewer than 4 posttest scores imputed 

AR analysis sample 

School n = 33 
Student n = 222 

Fewer than 4 posttest scores imputed 

N-AR analysis sample 

School n = 35 

Student n = 744 

54 posttest scores imputed 

N-AR analysis sample 

School = 29 a 

Student n = 701 

49 posttest scores imputed 


A4 




Planned grade 9 mathematics courses 

AR analysis sample 

AR analysis sample 

School n = 35 

School n = 33 

Student n = 218 

Student n = 222 

Fewer than 4 planned grade 9 course indicators 
imputed 

1 1 planned grade 9 course indicators imputed 

N-AR analysis sample 

N-AR analysis sample 

School n = 35 

School = 29 a 

Student n — 744 

Student n = 701 

47 planned grade 9 course indicators imputed 

49 planned grade 9 course indicators imputed 

Actual high school coursetaking 

AR analysis sample 

AR analysis sample 

School n = 35 

School n = 33 

Student « =218 

Student n = 222 

7 high school coursetaking indicators imputed 

6 high school coursetaking indicators imputed 

N-AR analysis sample: na 

N-AR analysis sample: na 


na is not applicable. AR is algebra ready. N-AR is not algebra ready. 

Note: To maintain the confidentiality of participants, measures with fewer than three missing data cases were suppressed for 
presentation purposes. 

a. Four control schools did not have any N-AR students (all grade 8 students were classified as algebra ready). 

b. Students who were not testable were designated so by their schools (because of special education status and the schools’ 
determination that appropriate accommodations on the study-administered assessments could not be feasibly provided). 

c. The response rate and missing rate calculations do not include students who were deemed not testable by their schools or 
whose parents withdrew them from the study (nonconsent). Missing rates (but not response rates) count missing students whose 
scores were cut because they spent less than five minutes on the test. Therefore, missing rates and response rates do not always 
sum to 100%. 

Student Samples 

In June 2008, before random assignment, schools were required to identify the students they 
considered ready for algebra. A common criterion for defining “algebra ready” was not 
established, and schools varied in their approach. It was not clear at the time of random 
assignment whether schools would systematically identify their high-achieving students as 
algebra ready or use other criteria. 

The first approach to determining whether the AR sample was higher achieving than the N-AR 
students and whether the two groups differed in other ways was to test for differences in their 
characteristics. Across treatment and control conditions, researchers used a two-level model that 
accounts for the clustering of students within schools and includes the blocking factors used for 
randomization. The results indicate the following: 

• A smaller percentage of AR students than N-AR students received free or reduced-price 
lunch. 

• A smaller percentage of AR students than N-AR students received special education 
services. 

• AR students’ scores on both the prior year’s state mathematics assessment and the study- 
administered pretest were significantly higher than those of N-AR students. 
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AR and N-AR students in the study schools did not significantly differ on other demographic 
factors (table A-l). 


Table A-l. Baseline Characteristics of AR and N-AR Students in Study Schools 


Characteristic 

AR 


N- 

AR 

/i-value 


Number 

Percent 

Number 

Percent 


Eligible for free or reduced-price lunch 

139 

32 

670 

46 

<0.001 

Receives special education services 

15 

3 

252 

17 

<0.001 

Has limited English proficiency 

15 

3 

43 

3 

0.843 a 

Female 

214 

49 

717 

50 

0.476 

Racial/ethnic minority 

29 

7 

75 

5 

0.369 


Number 

Mean 

Numbei 

Mean 


Mean grade 7 score on state mathematics 

437 

0.95 

1,403 

-0.24 

<0.001 

assessment (standardized) b 


(0.69) 


(0.86) 


Mean fall 2008 pretest score (Promise 

435 

349.87 

1,384 

312.60 

<0.001 

Assessment) 0 


(23.27) 


(27.23) 



AR is algebra ready. N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control). Values represent unadjusted means. Differences in school 
characteristics by sample were tested using a model that accounted for the clustered data structure and blocking used for 
randomization. Figures in parentheses are standard deviations. 

a. Model did not estimate when controlling for five state by size dummy blocking variables. Reported p-value represents a 
model that controls for state and two dummy indicators for medium and large schools rather than their interactions. 

b. State assessment scores were standardized by using the mean and the standard deviation of the test scores within each state, 
including only schools participating in the study. 

c. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source: Maine state department of education and Vermont supervisory unions, study records. 


To further examine the extent to which AR and N-AR students represent different populations, 
researchers examined density plots of the sample distributions on the prior year’s state 
mathematics assessment (figure A-2) and the pretest (figure A-3). Although there is clear overlap 
in the distributions, there is evidence to suggest that schools did identify their higher-achieving 
students as algebra ready. 
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Figure A-2. AR and N-AR Ready Distributions on Prior Year State Mathematics 
Assessment 


Prior Year State Math Assessment 


.2 


o 

-4 -2 0 2 4 6 

Standardized state mathematics achievement (Z scores) 

AR 
N-AR 

kernel = epanechnikov, bandwidth = 0.1608 


AR is algebra ready. N-AR is not algebra ready. 

Source: Maine state department of education and Vermont supervisory unions. 



Figure A-3. AR and N-AR Distributions on Pretest 


Study-Administered Pretest 

.02 


.015 


Density 

.01 


.005 


0 


200 250 300 350 400 

Mathematics achievement 

AR 
N-AR 

kernel = epanechnikov, bandwidth = 6.2143 


AR is algebra ready. N-AR is not algebra ready. 
Source: Study-administered pretest. 
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Estimates of Statistical Precision Based on Data Used in Analyses 

Minimum detectable effect sizes were calculated based on Bloom (2005, equation 8). As the 
intervention was designed to have a direct effect on students’ mathematics knowledge and skills 
and a minimum detectable effect size of 0.25 is considered necessary for an intervention effect to 
have “educational significance” (Bloom et al. 2008), the study team established a target effect 
size for the analysis of impact of the intervention on AR students of 0.25: 


P = proportion of sample schools allocated to the online algebra treatment (0.5). 

J = total number of schools in study sample. 

K = number of school-level covariates used. 

Tl = number of students per school at posttest and follow-up. 

P = school-level intraclass correlation. 

R c = proportion of the variance between schools that is reduced by the covariate 
(school-level explanatory power). 

r 2 = proportion of student-level variance component explained by student-level 

pretests. 

M = multiplier that translates the standard error into an estimate of minimum detectable 
effect size. It is equal to the t critical value for OC , the significance level of the 


Assumed Statistical Power for Impact Analyses with AR Students 

Calculations of the minimum detectable effect size for analysis of algebra achievement at the end 
of grade 8 are based on the following assumptions: 

Statistical power. 80%. 

Statistical significance level : Alpha of 0.025 for a two-tailed test, adjusted for multiple 
comparisons (two outcomes). 

Number of schools (actual): 68 . 

Number of students per school (actual): Each school served an average of 32 grade 8 students, 
6.54 (20%) of whom were identified as algebra ready. Power calculations were conducted 
assuming 88% response rates (that is, six AR students per school at posttest). 

Proportion of students in treatment condition: 50%. 



where 


intended statistical test, plus the t critical value for P , the likelihood of detecting 
significant effects given a true effect of a particular size (that is, the power of the 
test). 
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Covariate adjustment : 

School level : Researchers assumed that two-thirds of the school-level variance in the 
achievement outcome would be explained by the baseline mathematics scores. 54 They assumed 
that half of the school-level variance in the coursetaking outcome would be explained by the 
pretest. Two additional school-level covariates were the indicators for the two variables by which 
schools were blocked for random assignment: state and size. 

Student level : Researchers assumed that pretest scores and additional student-level covariates, 
including gender, eligibility for free or reduced-price lunch, and special education status, would 
explain 50% of the variance in posttest scores (that is, R 2 = 0.50). 

Intraclass correlation : Researchers assumed an intraclass correlation value of 0. 12. This value is 
based on Hedges and Hedberg’s 2007 empirical analysis of intraclass correlations using a 
nationally representative dataset from the Longitudinal Study of American Youth (LSAY). In 
that analysis, the unadjusted (unconditional) intraclass correlation for grade 8 mathematics for 
rural schools in the Northeast region of the country was 0. 12 (Hedges, personal communication, 
October 15, 2007). 

Blocking: Calculations of minimum detectable effect size assume blocking by two stratification 
variables: state (Maine, Vermont) and school size based on the total number of grade 8 students 
served (small: fewer than 17; medium: 17-70; large: more than 70). Power analyses assumed 
that random assignment of schools was conducted separately within each block, resulting in an 
equal number of intervention and control schools within each block. 

Based on these assumptions, researchers calculated a minimum detectable effect size of 0.25 for 
the algebra achievement outcome. Using a similar set of assumptions but a lower correlation 
between pretest and outcome, they calculated a minimum detectable effect size of 0.27 for the 
coursetaking outcome. 

Assumed Statistical Power for Impact Analyses for N-AR Students 

Calculations of the minimum detectable effect size for analysis of outcomes for the full N-AR 
sample were based on the same assumptions as for the full AR student sample, except for the 
number of students per school and the significance level. For secondary analyses involving the 
N-AR sample, researchers assumed an alpha of 0.05 for a two-tailed test of statistical 
significance, with no adjustment for multiple comparisons; an average of 22.9 N-AR students 
per school; and power calculations that assume 88% response rates at follow-up (20 N-AR 
students per school). Based on these assumptions, a minimum detectable effect size of 0.19 was 
calculated for the analysis of impacts on mathematics achievement (algebra and general 
mathematics) at the end of grade 8 for N-AR students and 0.22 for the analysis of impacts on 
coursetaking by N-AR student. 


54 The assumption that 66% of the variance in posttest scores would be explained by pretest scores is 
based on the correlation of school-level average scores on Maine’s mathematics assessment in grades 7 
and 8, which was 0.81. 
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APPENDIX B: MEASURES 


This appendix provides detail on the instruments and measurement strategies used in this study. 
These measures include measures of implementation derived from classroom observations, 
proctor logs, a teacher survey, and classroom materials. 

This appendix also provides detail on the outcome measures (Promise Assessment and coding 
protocol for planned high school courses). For each type of measure, it describes the 
development of instruments, data collection procedures, and analytic procedures, including scale 
construction. 

Implementation Measures 

Classroom Observations 

Study team members visited all treatment schools to observe how the online Algebra I course 
was being implemented. The following protocol items were completed in all 35 schools. 

Figure B-l. Protocol for Observing Implementation of Online Algebra I Class 


|General Information 


Observer name: 

Date and time of class period: 

School name: 

Proctor name: 

Proctor position at school: 

Where do students access the online algebra course? (Check one) 

□ Library 

□ Computer lab 

□ Classroom with regular 8th grade math class 

□ Classroom separate from regular 8th grade math class 

□ Other (please specify): 


Did the proctor or other staff member have other responsibilities or do other work 
during the visit? Yes No 


Proctor Logs 

The proctors who supervised the students in the online course completed a weekly web-based log 
to record the amount of time they engaged in different types of activities as part of their proctor 
role. The first set of questions asked proctors to report how much time, in minutes, they spent 
that week on the following activities: 

• Answering students’ questions about algebra 

• Answering students’ questions about nonalgebra mathematics 
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• Providing technical/computer support 

• Proctoring quizzes or tests 

• Communicating with the online teacher 

• Contacting technical support from Class.com 

• Contacting technical support from their school 

• Transferring student grades from the online course to their school’s grade book or 
grading system 

• Handling scheduling or other administrative issues 

• Addressing student behavioral problems. 


The second set of questions asked proctors to report the number of communications they had 
during the week with the online teacher, technical support at Class.com, and technical support at 
their school. 

Teacher Survey 

Researchers administered the teacher survey online in spring 2009 to all grade 8 mathematics 
teachers in study schools and to the online Algebra I teachers. The survey served two main 
purposes. First, it provided data on characteristics of the teachers (for example, degree earned, 
years of teaching experience). Second, it provided data on the organization and delivery of grade 
8 mathematics instruction in treatment and control schools, as well as in the online Algebra I 
classes. 

The data supplemented the analyses of class size and ability grouping. The teacher survey 
included an item that asked teachers to list the number of students enrolled in each section of 
grade 8 mathematics they taught. This item was used to calculate the average number of students 
per section at each school, which is reported in chapter 3 when comparing the class sizes in 
treatment and control schools. 

The teacher survey also included two items that asked teachers to describe the extent to which 
they provided different types of instruction based on the ability levels of students in their 
classrooms. One item asked teachers to describe the extent to which they differentiated their 
instructional practices in general. The other asked teachers to describe the extent to which they 
provided accelerated learning opportunities (such as Algebra I) for higher-ability students. These 
items were used to describe teachers’ instructional practices in chapter 3. 

Classroom Materials 

To estimate content coverage in the general grade 8 mathematics classes in study schools, the 
study team collected instructional materials, including pacing guides and course syllabi, textbook 
information/tables of contents, classroom assignments, and exams. While visiting schools during 
the spring 2009 posttest administration period, trained field staff used a protocol to collect 
relevant classroom materials from grade 8 classroom teachers. The protocol guided data 
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collectors to ask all teachers to name the textbooks they used and to submit a table of contents, 
annotated with the topics covered, so that the study team could determine how much of the 
curriculum/textbook each teacher completed. In addition, the protocol guided data collectors to 
ask teachers to submit more detailed infonnation if available, including example assignments or 
exams, pacing guides, or any other materials that described the content taught in their course. 

The study team collected course materials from 60 schools. The materials were coded on the 
basis of the type of material collected, the percentage of class time spent on algebra topics, and 
the textbooks used. Seven control schools offered a separate Algebra I course for certain grade 8 
students. Researchers computed a weighted average (described below) to measure the algebraic 
content focus at these schools. 

The study team received tables of contents from 50 schools (27 control schools, 23 treatment 
schools). Teachers who returned tables of contents that were not annotated were assumed to have 
completed all units listed. Researchers also collected 17 pacing guides and 12 annotated syllabi. 
Notes on pacing guides helped researchers determine the amount of time spent on each 
mathematics topic. When the team received a syllabus that was not annotated, it assumed that all 
topics had been completed. 

A code of 1-4 was given to each set of materials submitted (1 = 25% of time spent on algebra, 2 
= 50% of time spent on algebra, 3 = 75% of time spend on algebra, 4 = 100% of time spent on 
algebra). The coding was guided by the identification of algebraic terms that were consistent 
with the 1 1 items from the section on the teacher survey addressing algebra content. The coding 
process was conducted by two team members with experience reviewing mathematics curricula, 
including one senior mathematics expert. The junior team member first coded all 61 sets of 
materials. The senior mathematics expert then coded a random selection of 10 sets of classroom 
materials. The only area of disagreement involved one set of Connected Math materials, for 
which the coders differed by 1 (2 versus 3). Because Connected Math was the most common 
curriculum, the senior mathematics expert recoded these schools. For control schools that 
included both an Algebra I course and a general mathematics course, researchers computed a 
weighted school average based on the number of students assigned to each course within each 
school. For example, the weighted average for a control school in which 8 students took a 
separate Algebra I course coded 4 and 10 took a general grade 8 mathematic course coded 2 
would be [(8*4)+(10*2)]/18 or 2.89. With rounding, this school would be coded 3. 

The most common mathematics textbook/curriculum used in both treatment and control schools 
was Pearson-Prentice Hall Connected Mathematics (table B-l). Many of the units in this 
program focus on algebra content, covering such topics as patterns and sequences, linear 
equations and inequalities, algebraic expressions, and quadratic equations. However, the 
curriculum is not organized like that in a typical Algebra I textbook. Instead of following the 
chapter and single-day lesson structure of a typical Algebra I textbook, Connected Mathematics 
is organized into four- to five-week units that address a common mathematical theme or idea. 
Each unit focuses on the overarching mathematical idea through several multiday investigations 
that are often grounded in a real-world or practical context. Thus, although Connected 
Mathematics has a sharp focus on algebraic concepts, the concepts are presented in a fonnat 
different from that of a typical Algebra I textbook. Eighteen schools (9 control, 9 treatment) used 
this program. McDougal Littell Math Thematics was the second most popular text, with 13 
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schools (8 control, 5 treatment). The most common Algebra I materials were Holt Algebra 1 and 
Prentice Hall Algebra 1. Other common grade 8 mathematics texts included Glencoe 
Mathematics Course 3, Glencoe MathScape, Saxon Course 3 (Algebra Zi) and Forseman- Wesley 
Math. 


Table B-l. Mathematics Textbooks Used in Study Schools 


Textbook 

Pearson/Prentice Hall Connected 
Mathematics 
McDougal Littell Math 
Thematics 

Glencoe Mathematics Course 3 
Glencoe MathScape 
Saxon Course 3 ( Algebra A) 
Prentice Hall Algebra I 

Holt Algebra I 
Foresman-Wesley Math 


Source: Classroom materials. 


Website for more information 

http://www.phschool.cony cmp2/ 

http://holtmcdougal.hmhco.com/hm/math.htm 

http://www.glencoe.com/sec/math/msmath/mac04/course3/ 
http://www.glencoe.com/sec/math/mathscape/index.php/ 
http://saxonpublishers.hmhco.com/en/sxnm middle.htm 
http://www.phschool.com/atschool/ph algebra/program page, 
html 

http://go.hrw.com/gopages/ma/algl 07.html 
http://www.pearsonschool. com/index. cfm?locator=PSZu6e&P 
MDBSUBCATEGORYID=25741&PMDBSITEID=2781&PM 
DBSUBS0LUT10N1D=&PMDBS0LUTI0NID=6724&PMD 
BSUBJECTAREAID=&PMDBCATEGORYID=8Q6&PMDbP 
rogramld=5052&elementType=programComponents 


Number of 
schools 

18 

13 

8 

4 

4 

4 

4 

4 


Study-Administered Pretest and Outcome Measures 

The pretest, general mathematics posttest, and algebra posttest used in this study were computer- 
adaptive tests called the Promise Assessments, developed by SEG Assessment and Internet 
Testing Systems. The Promise Assessment has been administered to more than 200,000 students 
since 2007 to assess and diagnose mathematical strengths and weaknesses. 

The Promise Assessment battery is composed of two assessments: the pre-algebra mathematics 
assessment, a grade 3-8 assessment of mathematical skills, and an assessment of skills 
associated with elementary algebra (Algebra I). The pre-algebra mathematics assessment 
measures the range of mathematics content covered in grades 3-8, including number and number 
sense, computation and estimation, measurement, geometry, probability and statistics, patterns, 
functions and algebra. The Algebra I assessment measures content related to equations and 
inequalities, algebraic manipulation and graphing systems of equations, polynomials, quadratic 
equations, and functions and expressions. The two assessments may be used alone or together, 
depending on the assessment and the instructional needs of the user. 

The Promise Assessments are delivered online. The instrument determines which items to 
administer on the basis of each student’s response to previous items. In this way, the test assesses 
students’ skills in the shortest amount of time. 


For the fall implementation, the Promise General Mathematics Assessment was delivered as a 
30-item computer- adaptive test. For the spring implementation, the Promise General 
Mathematics Assessment was delivered as a 20-item computer-adaptive test and the Promise 
Algebra Assessment was delivered as a 20-item computer-adaptive test. 
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Development and Psychometric Properties of the Promise Assessment 

Approximately 2,500 test items were written to measure prealgebra skills in grades 3-8 and 
algebra. The test items were written to match the Virginia Standards of Learning 
(http://www.doe.virginia.gov/testing/sol/standards_docs/index.shtml). Those standard, developed 
to reflect the range of skills expected of students in grades 3-8 and in Algebra I, were based 
largely on other nationally developed mathematics standards. Approximately 15-20 test items 
align to each standard. Approximately 30% of the items included graphic stimuli, such as a 
geometric figure, a table, or a graph. Information on item development, field testing, item 
analysis, and calibration is available upon request from the test developer. 

Scaling and Scoring. One of the benefits of the linking and calibration design is the ability to 
place all items and individuals on a single underlying scale reflecting algebra readiness for the 
General Mathematics Assessment and a single underlying scale reflecting algebra skill for the 
Algebra Assessment. For each scale, the scores were initially calibrated on a scale ranging from 
-3 to +3 (with a mean of 0 and standard deviation of 1). 

The standard scale values were then linearly transformed. The General Mathematics Assessment 
was linearly transformed to a scale of 200-400, with a mean of 300 and a standard deviation of 
33. The Algebra Assessment was linearly transformed to a scale of 400-500, with a mean of 450 
and a standard deviation of 17. As linear transformations, these scales may be interpreted 
consistently with the underlying -3 to +3 standard scale. 

Reliability. Reliabilities of the Promise Assessment General Mathematics and Algebra tests are 
reported as the Rasch person reliability estimates for the nonning population on which the tests 
are scaled. The reliabilities reported by the test developer were 0.87 for the General Mathematics 
Assessment (standard 30-item test), 0.83 for the General Mathematics Assessment (as a 20-item 
test), and 0.80 for the Algebra Assessment (as a 20-item test). The study team calculated the 
reliability of the tests as used in this study on the sample of all grade 8 students in study schools 
who took each assessment (AR and N-AR combined). The estimated reliabilities of the scores 
were 0.83 for the general mathematics pretest, 0.77 for the general mathematics posttest, and 
0.71 for the algebra posttest. 

Standard Error of Measurement. The overall standard error of measurement was estimated as 
the standard deviation multiplied by the square root of the test reliability subtracted from 1 
(standard deviation x SQRT(1 - r xx )). The Rasch-based reliability estimate was used in 
calculating the standard errors of measurement, which were as follows: 

• 30-item General Mathematics Assessment: 12.71 scale points (200-400 scale) 

• 20-item General Mathematics Assessment: 15.56 scale points (200-400 scale) 

• 20-item Algebra Assessment: 8.93 scale points (400-500 scale). 

Validity. The developers of the Promise Assessment General Mathematics test claim that the 
scores measure a student’s level of prealgebra mathematics proficiency in relation to 
mathematics standards in grades 3-8. The Promise Assessment Algebra test purportedly 
measures a student’s level of algebra proficiency in relation to mathematics standards defining 
the typical Algebra I course. Validation is an ongoing process of collecting evidence in support 
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of the test and test scores. Several steps have been taken to establish the validity of the Promise 
Assessment. 

• The test items were developed by mathematics subject matter experts. 

• The test items were developed specifically to measure each of the standards for which 
the program was developed. 

• The test items were reviewed multiple times for content accuracy, alignment, and 
freedom from potential bias. 

• The test items were reviewed by mathematics educators on two separate occasions 
(online and face to face). 

• The test items were field-tested with more than 20,000 students. 

• The test items were calibrated and found to fit with the expected mathematical model 
(Rasch one-parameter model). 

• The test items were reviewed, and any item that failed to meet established 
psychometric criteria was eliminated. 

• The dimensionality of the test was reviewed and confirmed to be a unidimensional 
construct. 

• Students performed as expected on the field test, with the mean test results for each 
grade falling in an expected, ordered pattern (that is, grade 8 students perfonned 
better than grade 7 students, who performed better than grade 6 students, and so 
forth). 

• A simulation study of the computer-assisted test algorithm in which 100 tests were 
“taken” by simulated examinees at a range of ability levels found that the computer- 
assisted test produced scores as expected (table B-2). 

Table B-2. Results of Simulation Study of Computer-Assisted Test 


Simulated ability 
level 

Average observed ability level 
across 100 trials 

-2 

- 1.95 

-1 

- 1.01 

0 

- 0.01 

1 

0.96 

2 

2.04 


Source: Internet Testing Systems and SEG Assessment. 


Sample Items 

The sample items shown in this section denote the level of item difficulty at a particular scaled 
score. A student of ability level X is said to have a 50% probability of correctly answering an 
item at the same difficulty level. For example, approximately 50% of all students earning a 
scaled score of 200 would correctly answer the sample item shown for a scaled score of 200. The 
sample items provided are not official Promise Assessment items. 
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General Mathematics Assessment. The general mathematics assessment was scored on a scale 
of 200-400. Fifty percent of students with a score of 200 (a low score, associated with an ability 
level of -3.0) on the general mathematics posttest would correctly answer the following item: 


Sample Item 1: Scaled Score = 200 ord = -3.0 


Which statement is true? 


A. 

256 

> 

321 

B. 

517 

< 

541 

C. 

300 

= 

301 

D. 

472 

< 

468 

Correct 

response: B 


Fifty percent of students with a score of 300 (an average score, associated with an ability level of 
0.0) would correctly answer the following item, which is more difficult: 

Sample Item 2: Scaled Score = 300 or& = 0.0 

Which method can be used to solve the number sentence below? 

6 + x= 13 


A. Multiply each side by 6. 

B. Divide each side by 6. 

C. Add 6 to each side. 

D. Subtract 6 from each side. 

Correct Response: D 

A student with a score of 400 (a high score, associated with an ability level of 3.0) would 
correctly answer the following item, which is more difficult: 
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Sample Item 3: Scaled Score = 400 or 0 = 3.0 


A 20-foot ladder is leaning against a house. The foot of the ladder was placed 6 feet from the side of the 
house. About how high up the side of the house does the ladder reach ? 

A. 14 feet 

B. 19 feet 

C. 20 feet 

D. 21 feet 
Correct Response: B 

Algebra Assessment. The algebra mathematics assessment was scored on a scale of 400-500. 
Fifty percent of students with a score of 400 (a low score, associated with an ability level of- 
3.0) on the algebra posttest would correctly answer the following item: 


Sample Item 1: Scaled Score = 400 ord = -3.0 


-5(2x + 3) = 20 can be simplified to -lOx - 15 = 20 by using which property? 

A. transitive property of equality 

B. reflexive property of equality 

C. associative property of multiplication 

D. distributive property of multiplication 
Correct Response: D 

Fifty percent of students with a score of 450 (an average score, associated with an ability level of 
0.0), would correctly answer the following item: 


Sample Item 2: Scaled Score = 450 ord = 0.0 

Factor the expression 4x 2 + 20x + 25. 

A. (2x + 5)(2x - 5) 

B. (2x + 5) 2 

C. (2x - 5) 2 

D. (2x + 25)(2x + 1) 

Correct Response: B 
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Students with a score of 490 (a high score, associated with an ability level of 3.0), would 
correctly answer the following item: 


Sample Item 3: Scaled Score = 490 or 9 = 2.4) 


How many solutions does this system of equations have? 

3 7 

y = — x 2 - 5 
y +-x- 7 = 0 

A. no solutions 

B. infinite solutions 

C. only one solution 

D. only two solutions 
Correct Response: D 

Administration of Promise Assessment 

Collection of Pretest Data. In early fall 2008, the Promise Assessment was administered to all 
grade 8 students in schools participating in the study. The majority of students participating in 
the study (59%) completed the pretest before the end of September 2008. All pretesting was 
completed by mid-October 2008. In all schools, the pretest was administered by staff at the 
school, including grade 8 mathematics teachers, technology specialists, other teachers, and 
principals. 

Before the administration of the pretest, the study team collected full rosters of all grade 8 
students, including their names, their state identification (ID) numbers, and the names of each 
student’s mathematics class and teacher. The study team used these rosters to create materials for 
the pretest that included electronic templates to upload into the online assessment system; 
attendance rosters for study liaisons to log the completion date of the pretest for each student and 
note any accommodations made; and personalized, individual log-in sheets showing each 
student’s name and secure, study-specific ID. 

Each student logged in to the system and was routed to the pretest for the study. The test 
followed the procedures defined by Internet Testing Systems, the company that delivers the 
Promise Assessment, beginning with a brief tutorial on how to use the features of the 
computerized test. These features include selecting and changing answer choices, moving 
between items, using tools such as protractors, and ending the test. After students completed the 
tutorial, the first test item was presented. The difficulty of the test was adjusted to match each 
student’s performance, so each student saw different items. Item difficulty was based on whether 
a student gave correct responses to items up to that point. As a student answered correctly, the 
questions became more difficult; incorrect answers were followed by easier items. 

Variation in Time Spent on Test. Because the Promise Assessment is administered online 
through a web-based testing system, precise information on the amount of time spent by each 
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student on the test was available to the study team. These data indicated that there was school- 
level variation in the administration of the assessment. In particular, the study team identified a 
difference by condition in the amount of time, on average, spent on the pretest by both AR and 
N-AR students. Specifically, students in treatment schools spent more time on the test than 
students in control schools (table B-3). 


Table B-3. Average Time (in Minutes) Spent on Promise Assessment Pretest 


Student sample 

Treatment schools 
(standard deviation) 

Control schools 
(standard deviation) 

Difference 

p-value 

AR 

33.20 

(13.78) 

28.02 

(9.49) 

5.18 

0.039 

N-AR 

25.05 

(9.87) 

22.99 

(7.99) 

2.06 

0.038 


AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control); 436 AR students (217 treatment, 219 control); and 1,400 N-AR 
students (724 treatment, 676 control). Four control schools had no N-AR students. Data were missing for 4 AR students and 45 
N-AR students. Reported means, standard deviations, and statistics tests represent observed data (before imputation). 

Statistical tests comparing treatment and control students were performed on an imputed dataset; no changes in statistical 
significance emerged. The means and standard deviations reported are observed, not model adjusted; the /(-values are based on 
a model estimating the effect of treatment status on testing time, controlling for school-level state by size blocking variables. 
Source: Promise Assessment Pretest results. 


On average, AR students in treatment schools spent five minutes longer on the pretest than AR 
students in control schools. On average, N-AR students in treatment schools spent two minutes 
longer on the pretest than their counterparts in control schools. The differences were statistically 
significantly for both groups. 

The difference in testing time in treatment and control schools indicates a “late pretest problem” 
for this assessment of baseline mathematics achievement. It is possible that the reason for the 
testing time difference is that schools that were randomly assigned to the treatment condition 
were more aware of being in the study than schools assigned to control and that treatment 
schools, teachers, and students therefore took the pretest more seriously. 

All scores based on very low testing times (defined as less than five minutes) were excluded 
from analysis. There were no such scores among AR students on the pretest; among the N-AR 
sample, 16 scores were cut for this reason (10 in treatment schools and 6 in control schools). 

Collection of Posttest Data. The study team implemented a similar procedure for the posttest 
assessment, except that trained members of the study team visited each school to administer the 
test. The purpose of this change was to standardize administration across schools for the 
measurement of the key achievement outcomes at the end of grade 8. The study team arranged a 
site visit with each school to administer the posttest. The same instructions were given to 
students in all schools. In all cases, students first received the 20-item general mathematics 
posttest and then received the 20-item algebra posttest. 

There was not a significant difference in testing time on the algebra portion of the posttest for 
AR students. However, a sensitivity analysis was conducted to ensure that testing time on this 
primary outcome (algebra posttest scores) did not account for the observed treatment effect (see 
appendix E, model E.7). As with the pretest, there was a significant difference by condition in 
the amount of time spent on the general mathematics posttest by AR students (table B-4): on 
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average, AR students in treatment schools spent nearly four minutes longer than AR students in 
control schools. There were no significant differences in testing time by condition on either 
posttest for N-AR students. 


Table B-4. Average Time (in Minutes) Spent on Promise Assessment Posttests 




Treatment 

Control 



Posttest 

Student sample 

schools 

schools 

Difference 

p-value 


AR 

N-AR 

17.69 

15.87 

1.82 

0.139 

Algebra Posttest 

(6.85) 

12.76 

(6.00) 

12.22 

0.54 

0. 1 59 a 


(6.46) 

(5.36) 




AR 

21.85 

18.31 

3.54 

0.002 

General Math 

(8.60) 

(6.43) 




N-AR 

17.62 

16.70 

0.92 

0.161 


(7.90) 

(6.65) 




AR is algebra ready. N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control); 436 AR students (217 treatment, 219 control); and 1,400 N-AR 
students (724 treatment, 676 control). Four control schools had no N-AR students. Data were missing for 4 AR students and 45 
N-AR students. Reported means, standard deviations and statistics tests represent observed data (before imputation). Figures in 
parentheses are standard deviations, 
a. p - value became significant (.047) in the imputed dataset. 

Source: Promise Assessment Pretest results. 


Scores based on testing times of less than five minutes were excluded from analysis. There were 
no such scores among AR students on the general mathematics posttest; there were fewer than 
four such scores among AR students on the algebra posttest. Among the N-AR sample, 17 
scores were cut because of low testing time on the general mathematics posttest (13 treatment, 4 
control), and 118 were cut on the algebra posttest (73 treatment, 45 control). The large number of 
students who completed the test in less than five minutes may indicate that the test was difficult 
for N-AR students who did not take an Algebra I course in grade 8. 

High School Mathematics Courses 

The coding of the transcript data for impact analyses was based on methods used by the National 
Center for Education Statistics for the National Assessment of Educational Progress and 
Education Longitudinal Study transcript studies (U.S. Department of Education 2007a; 2007b). 
Transcript coding protocols guided the extraction of course identifiers. Mathematics education 
experts coded the course titles using the Classification of Secondary School Courses, which is 
based on information available in school catalogs and other information sources (U.S. 

Department of Education 2007b). The study team used the procedures described below to collect 
and code high school courses, including planned grade 9 courses (at the end of grade 8 for all 
students) and grade 9 and 10 course information (at the end of grade 9 for AR students). 

Planned Grade 9 Courses 

While visiting schools during the spring 2009 posttest administration period, trained field staff 
collected information on planned grade 9 courses from participating schools. The study team 
trained the field staff in April 2009, covering all aspects of spring data collection, including 
posttest and survey administration, collection of classroom materials, and collection of the 
planned grade 9 course information for all students in the school. 
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In many schools, students had not yet registered for their high school courses. In these cases, the 
information was collected at a later date by following up with the middle school study liaison by 
email or telephone. The school infonnation was defined as the source for planned grade 9 
courses. Continued verification of the course information occurred through summer 2009. 

Each planned grade 9 course was coded in a two-phase process. Phase one involved assignment 
of codes, based on a scheme based on the Classification of Secondary School Courses 
Hierarchical Mathematics Course Listing: 

1 = course below Algebra I (Pre- Algebra, Algebra I A) 

2 = Algebra I (or the second half of an Algebra I course [for example, Algebra I B]) 

3 = Algebra I Honors (includes Advanced, Accelerated) 

4 = course above Algebra I (for example, Algebra II, Geometry, Calculus). 


Throughout the coding process, the study team took steps to ensure that the data provided by the 
schools were reviewed carefully and coded consistently. The study team accomplished this by 
double coding, checking for interrater reliability, and using the study’s mathematics content 
experts to resolve discrepancies. In addition, it examined grade 9 course information from the 
high schools (for example, course catalogs, course descriptions on the web) and contacted 
relevant personnel from middle or high schools in a few cases. 

All courses with the standard titles Pre -Algebra, Algebra I, Geometry, and Algebra II were coded 
by two junior members of the study team. Courses with different titles (for example, Algebra IA, 
Integrated Mathematics II) were coded by two senior content experts from the study team. If the 
course titles were ambiguous for the content expert, the junior team members conducted web 
searches and found syllabi and course catalogs to provide clarification. One of the senior 
mathematics experts first coded all the courses with ambiguous titles. Any course titles that were 
unclear were flagged and then double-coded by the second mathematics content expert. There 
were 66 such cases. The two experts independently agreed on 62 of the 66 cases and came to 
resolution on the remaining cases via discussion. 

After Phase 1 coding was complete, planned courses were collapsed into three categories: 

1 = course below Algebra I in the Classification of Secondary School Courses sequence 

2 = Algebra I (includes Honors and Accelerated Algebra courses) 

3 = courses above Algebra I in the Classification of Secondary School Courses sequence. 


AR students’ planned coursetaking was categorized as advanced or not advanced. The analysis 
tested the impact of online Algebra I on the likelihood of enrolling in an advanced grade 9 
course. N-AR students’ planned coursetaking was categorized as being intermediate or not 
intermediate, where an intennediate sequence begins with Algebra I in grade 9. This lower bar 
for the N-AR student sample was considered to be appropriate given that they were not 
identified as ready for algebra as rising grade 8 students. 

To analyze the likelihood of AR students’ initially enrolling in an advanced course in grade 9, 
the study team collapsed the Phase 2 codes into a binary variable: 
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0 = course with a Phase 2 code of 1 or 2 (Algebra I or lower) 

1 = course with a Phase 2 code of 3 (above Algebra I). 


The analysis of this outcome is ancillary to the main high school coursetaking outcome based on 
follow-up data for the AR sample. It is reported as an exploratory analysis in chapter 5. 

To analyze N-AR students’ likelihood of following an intermediate course sequence in grade 9 
(as indicated by enrollment in an Algebra I course rather than a lower-level course), the study 
team collapsed the Phase 2 codes into a second binary variable: 

0 = course with a Phase 2 code of 1 (course lower than Algebra I) 

1 = course with a Phase 2 code of 2 or 3 (course at or above Algebra I). 


The analysis of this outcome addresses one of the secondary research questions for the N-AR 
sample and is reported in chapter 4. (No additional follow-up information on N-AR students’ 
coursetaking was collected.) 

Actual Grade 9 Courses 

During the spring 2009 data collection (when students were at the end of grade 8), researchers 
collected information on which high school each student planned to attend the following fall. In 
fall 2009 (when students began grade 9), the study team began preparing for high school course 
data collection by assigning one study liaison to each of the high schools AR students planned to 
attend. To the high school principals or mathematics department chairs, the liaisons described the 
goals of the study, identified key contacts for future data collection, discussed how the spring 
2009 data would be collected, and detennined whether the AR students who planned to attend 
each high school were actually enrolled. If students had transferred from the planned high school 
to another high school, researchers asked for the name of the new high school and contacted the 
new high school. For AR students who never attended the planned high school, researchers 
followed up with the middle schools to find out where the students ultimately enrolled. A total of 
66 high schools received AR students in the 2009/10 school year. 

The liaisons also collected course catalogs and any other information the high schools could 
provide about the types of mathematics courses they offered and how they fit into the school’s 
mathematics course sequence or sequences. The study team reviewed these materials to ensure 
that it had sufficient infonnation to code the courses. The liaisons followed up with any schools 
for which additional data were needed. From the course infonnation for each high school, 
researchers built a database that included the names of the courses, the credit structure, grading 
scales, and descriptions of each course (for example, prerequisites, key mathematics content 
covered). The database was completed before data collection began in May 2010 — the earliest 
date by which any of the high schools had given final course grades. 

Liaisons worked with high schools to establish the most efficient and reliable way to collect the 
course data. The primary way in which data were entered was through an online system that 
included all the information from the database previously described. The liaison entered the 
information in one of two ways. The first way was by calling a representative from each high 
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school and entering the data as the representative provided the infonnation over the telephone. 
The liaison then asked the high school representative to view the information that had been 
entered online and then to confirm the accuracy of the information by clicking a confirmation 
button. The second way was for the high school representative to enter the information directly 
into the system or complete an electronic template provided by the study team that included all 
the coursetaking variables (name of course, grade, and so forth). 

As with the planned grade 9 course data, the Classification of Secondary School Courses 
hierarchy was used to code the grade 9 and 10 course data. The grade 9 course titles and course 
grades and the planned grade 10 course titles were used to create the following composite 
coursetaking outcome, representing participation or nonparticipation in an advanced course 
sequence. The composite binary code was based on the following components: 

• Grade 9 mathematics course types. The codes for grade 9 course types are the same as 
those for planned grade 9 courses: 1 = course above Algebra I, 0 = course at or below 
Algebra I. 

• Grade 9 mathematics grades. Grades earned by AR students in their grade 9 
mathematics course were converted into a binary measure of “success” versus “not 
success”: 1 = grade equivalent of A, B, or C (success), 0 = grade equivalent of D or F 
(not success). If students took more than one mathematics course in grade 9, the grade 
associated with the more advanced course taken was used. 

• Grade 10 mathematics course types. The type of mathematics course in which each AR 
student enrolled for grade 10 was coded on the basis of the Classification of Secondary 
School Courses into the following categories: 1 = Algebra II or other course above 
Geometry in the typical course sequence, 0 = Geometry or other course below Algebra II 
in the typical course sequence. 

Composite Measure of Advanced Coursetaking 

The composite measure is a simple binary measure that represents whether students participated 
in an advanced mathematics course sequence in high school. Researchers assigned a 1 to all AR 
students who were assigned a 1 for all three of the component coursetaking measures (that is, the 
composite coursetaking code = 1 if the grade 9 course type = 1 and the grade 9 course grade = 1 
and the grade 10 course type = 1; otherwise, the composite coursetaking code = 0). The 
composite measure is a binary variable that signifies participation or nonparticipation in an 
advanced sequence of mathematics courses with a grade of at least C. 

Most high schools followed one of two sequences: Algebra I -> Geometry-^ Algebra II or 
Algebra I Algebra 1 1 -> Geometry, using these course titles. In these cases, course progressions 
were easy to classify as advanced or not advanced. Some course titles and sequences were less 
clear. For example, some schools used programs that integrated Algebra I, Geometry, and 
Algebra II content and offered accelerated versions of these courses. These cases required a more 
careful review of the course descriptions in the high school course catalogs. Where necessary, 
study team members called high school mathematics educators to clarify and confirm their 
interpretations of their course sequences. 
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Researchers began the coding process by creating a file that included the relevant course 
information for each student. (This file did not include any information that could be used to 
identify the treatment status of any students in the dataset.) Almost all the data were double- 
coded by two study team members with expertise coding high school courses, including one 
senior mathematics content expert. The same two coders led the coding of the planned high 
school courses in 2009, making them familiar with the content and process. The only data that 
were not double coded were cases in which the junior coder was unclear. These cases — 
approximately 10% of the sample — were then reviewed and discussed by the junior coder and 
the senior content expert in light of course catalog descriptions and additional information from 
high schools where necessary. 


B15 



APPENDIX C: INTERVENTION FEATURES 


This appendix describes the features of the online Algebra I course offered as the intervention in 
this study. 

Online Algebra I Course Content 

Algebra IA 

Algebra IA is designed to be equivalent to the first semester of a traditional middle/high school 
Algebra I course. This course covers, algebraic concepts, including integers, linear equations, 
linear inequalities, absolute value, polynomials, and factoring. 

Course Content 

• Unit 1 : Introduction to Algebra 

o Symbols in Algebra 
o Properties and Sentences 
o Real Numbers and the Number Line 

• Unit 2: Introduction to Equations 

o Combined Operations 
o Introduction to Function and Equations 
o Solving Equations Analytically 

• Unit 3: Solving Equations 

o Equations and Problem-Solving 
o Percentages 

• Unit 4: Inequalities and Absolute Value 

o Inequalities 
o Absolute Value 

• Unit 5: Polynomials 

o Monomials 
o Combining 
o Factoring 


Cl 


Course Objectives 


• Develop the fundamental algebraic skills of factoring polynomials, simplifying and 
evaluating expressions, and solving equations. 

• Apply algebraic problem-solving strategies to real-world situations. 

• Use graphing technology to interpret and solve equations and inequalities. 

• Communicate mathematical ideas, analyze mathematics situations, explain procedures for 
correct computation, and describe results using graphical, numerical, or algebraic 
representations. 

• Use the Internet to obtain useful infonnation. 

• Develop a sense of class membership, using newsgroups and Moodle learning 
management system to communicate with teacher and classmates. 

The following list includes all scorable activities and assessments available as part of Algebra 
IA. (As with a traditional face-to-face course online teachers had the option of “turning off’ 
particular activities such as within-unit exercises.). 

• 13 quizzes: At the end of each lesson in the course, students take a computer-graded quiz 
covering topics covered in the lesson. Students have one chance to take each quiz. Quiz 
results are automatically stored in the online environment grade report. 

• 5 unit exams: At the end of each unit in the course, students take a computer-scored unit 
exam covering all lessons in the unit. Students have one chance to take the exam. Unit 
exam results are automatically stored in the online environment grade report. 

• 1 final exam: At the end of the course, students take a two-part computer-graded final 
exam that covers topics from the entire course. Students have one chance to take the 
exam. Exam results are automatically stored in the online environment grade report. 

• 5 assignments: Each unit includes one assignment, which is completed online. Each 
assignment was created as part of the online quiz system. Teachers and proctors, but not 
other students, are able to see student responses. The assignments include open-ended or 
guiding questions that are not computer graded and not automatically included in the 
online environment grade report. 

• 5 discussions: Each unit includes one discussion assignment. The discussions were 
created as a forum in the online environment; they allow student responses to other 
students’ work and can include multiple threads. Teachers, proctors, and all students in 
the section can see the entire discussion. The discussions include open-ended questions 
that are not computer graded and not automatically included in the online environment 
grade report. 

Algebra IB 

Algebra IB is designed to be equivalent to the second semester of a traditional middle/high 
school Algebra I course aligned to the Maine and Vermont content standards. In this course, 
students continue their progression through algebraic concepts, expanding their knowledge of 
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functions and relations, solving systems of equations and inequalities, simplifying rational and 
radical expressions, and solving quadratic equations. A unit on probability and statistics is also 
included to help students analyze data and make predictions about real-world situations using a 
variety of visual representations. 

Course Content 

• Unit 1 : Linear Equations 

o Functions and Relations 
o Graphs and Linear Equations 
o Other Forms of Linear Equations 
o Variation 

• Unit 2: Systems of Equations and Inequalities 

o Solutions by Graphing 
o Solving Systems of Equations 
o Problem Solving 

• Unit 3: Rational Expressions and Radicals 

o Rational and Irrational Numbers 

• Unit 4: Quadratic Equations 

o Solving Quadratic Equations 
o Quadratic Equations and Problem Solving 

• Unit 5: Statistical Analysis 

o Measure of Variability 
o Probability 

• Unit 6: Variation, Rational Expressions, and Radicals 

o More Operations on Rational Expressions 
o Operations with Radicals 

• Supplemental Topic: Additional Statistical Analysis 

o Organizing Data and Variability 

Course Objectives 

• Add, subtract, multiply, and divide rational expressions. 

• Solve and graph linear functions. 

• Solve systems of equations and inequalities. 

• Perform operations with radicals. 
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• Solve radical equations. 

• Solve quadratic equations. 

• Analyze and interpret data represented in various real-world situations. 

• Apply algebraic problem-solving strategies to real-world situations. 

• Use graphing technology to solve, interpret, analyze, and compare linear functions and 
relations, inequalities, absolute value graphs, and quadratics. 

• Communicate mathematically by expressing ideas, analyzing situations, explaining 
procedures for correct computation, and describing results numerically and graphically. 

• Use the Internet to obtain useful information. 

• Develop a sense of class membership, using newsgroups, discussions, and email to 
communicate with teacher and classmates. 

The following list includes all possible scorable activities and assessments available as part of 
Algebra IB. 

• 17 quizzes: At the end of each lesson in the course, students take a computer-graded quiz 
covering topics in that lesson. Students have one chance to take each quiz. Quiz results 
are automatically stored in the online environment grade report. 

• 6 unit exams: At the end of each unit in the course, students take a computer-graded unit 
exam covering all the lessons in that unit. Students have one chance to take the exam. 
Unit exam results are automatically stored in the online environment grade report. 

• 1 final exam: At the end of the course, students take a two-part computer-graded final 
exam covering topics from the entire course. Students have one chance to take the exam. 
Exam results are automatically stored in the online environment grade report. 

• 6 assignments: Five units include at least one assignment; Unit 3 includes two 
assignments. Each assignment was created as part of the online quiz system and is done 
online. Teachers and proctors, but not other students, are able to see student responses. 
The assignments include open-ended or guiding questions that are not computer graded 
and are not automatically included in the online environment grade report. 

• 6 discussions: Four units include one discussion, Unit 3 includes two discussions, and 
Units 5 and 6 do not include any discussions. The discussions were created as a forum in 
the online environment; they allow student responses to other students’ work and can 
include multiple threads. Teachers, proctors, and all students in the section can see the 
entire discussion. The discussions include open-ended questions that are not computer 
graded and not automatically included in the online environment grade report. 

Alignment of the Class.com Online Algebra I Course with Two Typical 
Algebra I Textbooks 

Table C-l presents the percentage of lessons in Class.com Algebra IA and IB, the Algebra: 
University of Chicago School Mathematics Project textbook (McConnell 2002), and the Glencoe 
Algebra I textbook (Holliday et al. 2005) devoted to 1 1 typical Algebra I topics. These two 
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textbooks were chosen for alignment purposes because they are widely used in standard Algebra 
I courses in the region and across the country. 


Table C-l. Percentage of Lesson Content Devoted to Various Algebra I Topics by 
Class.com and Two Standard Textbooks 


Topic 

Class.com 
Algebra I A 
and IB 

University of 
Chicago School 
Mathematics 
Program 
Algebra 

Glencoe Algebra 

Patterns and sequences (arithmetic, geometric) 

2.8 

4.0 

3.3 

Simplifying and evaluating algebraic expressions 

19.4 

17.2 

14.3 

Functions (function notation, identifying functions) 

5.6 

9.1 

2.2 

Solving linear equations 

11.1 

7.1 

14.3 

Solving linear inequalities 

2.8 

3.0 

2.2 

Graphing linear equations (intercept and point-slope 
form of linear equations) 

8.3 

10.1 

11.0 

Graphing linear inequalities 

2.8 

2.0 

1.1 

Solving systems of linear equations and inequalities 

11.1 

8.1 

5.5 

Solving quadratic equations (factoring, quadratic 

11.1 

8.1 

11.0 

formula) 

Graphing quadratic equations 

2.8 

4.0 

3.3 

Solving and graphing other types of equations 
(polynomial, square root, absolute value) 

2.8 

5.0 

4.4 

Other 

19.4 

22.2 

27.5 


Note : All 36 lessons in the Class.com Algebra IA and IB course were included in this analysis. 

Source : Study team analysis based on review of topics and lessons in Class.com course and the UCSMP and Glencoe Algebra I 
textbooks. 
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APPENDIX D: ESTIMATION METHODS AND HYPOTHESIS 
TESTING 


This appendix describes the estimation methods used in and the hypothesis testing conducted for 
the study. 

Confirmatory Impact Analyses 

Primary Impact Analyses 

The analytic strategy was to compare students from schools randomly assigned to receive the 
intervention with students from schools that do not receive the intervention. Because treatment 
groups are determined at the school level, the primary unit of analysis is the school. The data for 
this study are hierarchical (students are nested within schools); therefore, units at the same level 
are not statistically independent. Ignoring the nested structure of the data yields misleadingly 
small standard errors for treatment effect estimates (see Seltzer 2004; Raudenbush and Bryk 
2002; Snijders and Bosker 1999). Given the hierarchical data structure in a cluster randomized 
setting, this study used hierarchical linear modeling to estimate the treatment effect on the 
outcomes of interest. Analyses using hierarchical linear modeling allow the effects of student- 
and school-level factors to be modeled and adjustments made for the nonindependence of 
observations within clusters. In all analyses, researchers estimated two-level models, where 
students were the Level 1 unit and schools the Level 2 unit. 

To handle missing data in the AR sample, researchers used multiple imputation with chained 
equations. All analyses of the AR sample were conducted with 10 multiply imputed datasets; the 
estimates presented in this appendix are averages based on the 10 datasets. For more information 
on the multiple imputation of missing data, see appendix F. 

Chapter 4 presented summarized results including model-based group means (for achievement 
outcomes) and predicted probabilities (for coursetaking outcomes) along with /7-values for the 
effect of condition. The results presented here are the coefficients and standard errors for all of 
the variables in each model, including student- and school-level covariates. The full-model 
coursetaking results additionally include corresponding odds ratios and 95 percent confidence 
intervals for each predictor. 

Primary Research Question 1 . To address Research Question 1, researchers calculated the 
intent-to-treat estimate of the effect of treatment status, using algebra posttest scores (Promise 
Assessment algebra score) at the end of grade 8 as the outcome. 

The following two-level model was specified: 


Algebraij = |3q/ + (3 y (StateMathy - StateMath . .) + ^(SpEdy - SpEd . .) + fiy(FRLy - FRL . .) 
+ \^ 4 j( Female ,j - Female..) + s,y (D. la) 
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Po/ = Yoo + Yoi TRTj + Y 02 ( MaineMEDj - MaineMED .) + y 0 3 ( MaineLGj - 
MaineLG .) + Y 04 ( VermontSMj - VermontSM .) + y 0 5 ( VermontMED ) - 


VermontMED.) + yo6 ( VermontLGj - VermontLG.) + Uq, (D. lb) 

Pi/ = Yio (D.lc) 

p 2 / = y 2 o (D.ld) 

P 3 / = Y 30 (D.le) 

04/ = 740 (D.lf) 


where Algebra y is the algebra achievement score for student i in school j at the end of grade 8 
(that is, the posttest) and StateMathy is the prior state mathematics score (standardized) for 
student i in school j. In the Level 1 model presented in equation D.la, all covariates are centered 
on the grand mean, so that Pq,- represents the adjusted mean for school j. 

The intent-to-treat estimate is specified in the first equation of the Level 2 model (equation 
D.lb). TRTj is an indicator variable that takes a value of 1 for treatment schools and 0 for control 
schools; yoo represents the average algebra achievement score for control schools; and yoi 
captures the difference in algebra achievement score between treatment schools and control 
schools (that is, the impact of the treatment when all other covariates are centered at the grand 
mean). The estimate yoi is reported as a standardized effect size, based on the pooled standard 
deviation for the treatment and control groups on the algebra posttest. A pooled standard 
deviation that accounts for both within and between imputation variance (Rubin 1987; Shafter & 
Graham 2002) was used to calculate an effect size across 10 imputed datasets. 

The model includes the following student-level covariates (centered on the grand mean): 

• Prior state mathematics assessment scores ( StateMath ), standardized within state using 
sample data only 

• Eligibility for free or reduced-price lunch ( FRL ) eligibility status (dummy variable coded 

0 = not eligible, 1 = eligible) 

• Eligibility for special education services ( SpEd) (dummy variable coded 0 = not eligible, 

1 = eligible) 

• Gender (dummy variable coded 0 = girls, 1 = boys). 

At Level 2, five dummy variables that capture state by size interactions were included, with 
small schools in Maine serving as the reference block: 


• Medium-size schools in Maine versus all other schools {MaineMED) (dummy variable 
coded 1 for medium-size schools in Maine, 0 for all other schools). 

• Large-size schools in Maine versus all other schools {MaineLG) (dummy variable coded 
1 for large-size schools in Maine, 0 for all other schools). 

• Small-size schools in Vermont versus all other schools {VermontSM) (dummy variable 
coded If or small-size schools in Vennont, 0 for all other schools). 
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• Medium-size schools in Vennont versus all other schools ( Vermont MED) (dummy 
variable coded 1 for medium-size schools in Vermont, 0 for all other schools). 

• Large-size schools in Vennont versus all other schools ( VermontLG ) (dummy variable 
coded lfor large-size schools in Vermont, 0 for all other schools). 

These student- and school-level variables were used as covariates in all analytic models for this 
study to improve the precision of the estimates of impact (table D-l). Variables were selected 
that were expected to be highly correlated with the outcome measure. 

Table D-l. Results of Main Impact Model Predicting AR Students’ Algebra Posttest Scores 


Variable 

Coefficient 

Standard error 

/?-value 

Effect size 

Intercept 

441.64 

1.11 

<0.001 


School covariate 

Condition 

5.53 

1.57 

0.001 

0.40 

Medium-size school in Maine 

1.48 

1.96 

0.454 


Large-size school in Maine 

2.55 

2.73 

0.354 


Small-size school in Vermont 

4.73 

3.11 

0.133 


Medium-size school in Vennont 

2.49 

3.50 

0.479 


Large-size school in Vennont 

3.14 

3.06 

0.311 


Student covariate 

Female 

0.09 

1.13 

0.939 


Receives special education services 
Eligible for free or reduced-price 

-11.18 

3.54 

0.002 


lunch 

State mathematics score 

1.98 

1.30 

0.129 


(standardized) 3 

9.87 

1.18 

<0.001 



Variance 

Chi-squared (degrees 
of freedom) 

/?-value 


Residual 

127.59 




Level 2 (school) 

10.67 

87.71 (61) 

0.014 


Total variance 

138.26 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating in 
the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Researchers conducted six sensitivity analyses that test the robustness of the observed estimates 
from the benchmark impact model. These analyses test models with different methods of 
accounting for students’ baseline mathematics achievement, a model with no covariates, a model 
based on observed (nonimputed) data only, and a model that tests an alternative nesting structure 
(with students in treatment schools clustered within online teachers). The results of these 
sensitivity analyses of the algebra outcome are reported in appendix E (models E.1-E.6). 

Primary Research Question 2. To addresses Research Question 2, the study team calculated the 
intent-to-treat estimate of the effect of treatment status, using participation in an advanced 
mathematics course sequence as the outcome. 

The following two-level hierarchical generalized linear model was used to examine the 
difference between the probability of participating in an advanced mathematics course sequence 
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in high school by AR students in treatment schools and control schools. Given a Bernoulli 
sampling model and a logit link function, the study team specified the following Level 1 model: 


T| ij = p 0/ + P y (StateMathj - StateMath..) + P 2 / SpEdj - SpEd..) + fiy(FRLj - FRL..) + 

p 4 [Female] - Female..) (D ,2a) 

where % = log (cp,y / 1 - cp,y) (that is, the log of the odds of participating in an advanced 
mathematics course sequence) and cp (/ is the probability of advanced coursetaking for student i in 
school j. If the probability of advanced coursetaking cp (/ is 0.5, the odds of advanced coursetaking 
r\ij = log ( cp,y / 1 - (p,y) = 0.5 1.5 = 1.0, and the log-odds (or logit) is log (1.0) = 0. If the probability 
of advanced coursetaking is less than 0.5, the odds are lower than 1.0, and the logit is negative. 

StateMathij is the (standardized) prior state mathematics score for student i in school j. In the 
Level 1 model presented in equation D.2a, all covariates are centered on the grand mean. The 
Level 1 model estimates the difference in log odds of participating in an advanced mathematics 
course sequence in high school by AR students from treatment schools and control schools, 
holding prior state mathematics assessment scores constant. 

The Level 2 models (including the same school-level variables as the previously specified model 
for algebra achievement at the end of grade 8) are as follows: 

Pqy = Yoo + Y 01 TRTj + Y 02 ( MaineMEDj - MaineMED.) + Y 03 ( MaineLGj - 
MaineLG.) + Y 04 ( VermontSMj - VermontSM .) + Y 05 ( VermontMEDj - 


VermontMED.) + Y 06 ( Vermont LGj - VermontLG.) + uq/. (D.2b) 

Pi/ = Yio. (D.2c) 

P2/ = Y 20 . (D.2d) 

P 3 / = Y 30 . (D.2e) 

p4/ = Y40. (D.2f) 


The key parameter of interest is Y 01 , the overall difference in log odds of participating in an 
advanced mathematics course sequence in high school between AR students in treatment schools 
and control schools. Unit-specific estimates are presented for all categorical models. 

The descriptive results for advanced coursetaking indicate that among AR students, 53% from 
treatment middle schools and 30% from control middle schools were participating in an 
advanced course sequence in high school (table D-2). 
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Table D-2. Percentage of AR Students Taking Advanced Mathematics Course Sequence in 
High School 


High school mathematics course sequence 

Students from 
treatment schools 

Students from 
control schools 

All AR students 

Advanced 

53.3 

29.5 

41.3 

Not advanced 

46.7 

70.5 

58.7 


AR is algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Percentages are 
calculated across 10 multiply imputed datasets. Coursetaking patterns were coded as representing successful completion of a 
course above Algebra I in grade 9 and enrollment in Algebra II or a higher course in grade 10) or not. 

Source: Coursetaking data collected from high schools study participants attended in 2009/10. 


Table D-3 shows the number and percentage of students by condition that met each criterion for 
participation in advanced mathematics course sequence in high school. 


Table D-3. Number and Percentage of AR Students Taking Advanced Mathematics Course 
Sequence in High School 


Criterion 

Total students 
(« = 427) 

Number Percent 

Students from 
treatment schools 
(« = 211) 

Number Percent 

Students from 
control schools 
(n = 216) 

Number Percent 

/>-value a 

l)Took course above Algebra I 
in grade 9 (n = 427) 

190 

45 

124 

59 

66 

31 

0.005 

2) Passed grade 9 course with 
grade of C or higher (/? = 425) 

392 

92 

194 

92 

198 

92 

0.974 

3) Enrolled in Algebra II or 
above for grade 10 (/? = 422) 

239 

57 

132 

63 

107 

50 

0.034 

Met all three criteria and were 
coded as advanced 

176 

41 

114 

54 b 

62 

29 

0.007 c 


Note: Sample includes 427 students (observed data). The sample size of 427 is different from the analytic sample of 440 
because this table represents a sensitivity analysis using only observed and not imputed data. 

13 students were missing data on criterion 1 (7 from treatment schools, 6 from control schools); 15 students were missing 
data on criterion 2 (8 from treatment schools, 7 from control schools); 18 students were missing data on criterion 3 (9 from 
treatment schools, 9 from control schools). 

For the second criterion, the grade 9 course refers to any mathematics course, including pre-Algebra and Algebra I. 

a. Significance testing was conducted using a two-level hierarchical linear modeling model with students at Level 1 and 
schools at Level 2. These models included all standard covariates included in the impact models and each criterion as a 
separate outcome variable. 

b. Percentage of AR treatment students who met all three criteria was 53% when averaged across the 10 multiply imputed 
datasets, however in the 427 observed data points (shown here), the correct percentage is 54%. 

c. Corresponds to impact model 

Source: Coursetaking data collected from the high schools study participants attended in 2009/10. 


Differences in predicted probabilities are presented in the main report to describe the differences 
in average predicted probabilities between students enrolled in treatment middle schools versus 
students enrolled in control middle schools (see table 4-3). These probabilities are the average, 
model-predicted probabilities, controlling for all covariates specified for the model. The logit 
estimates for the treatment and control groups were converted to average probabilities for each 
group. These group means and associated standard errors are generated in SAS using PROC 
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GLIMMIX by requesting the LSMEANS output for the contrast for condition where treatment = 
1 and control = 0. 


Table D-4 displays the complete results of the benchmark impact analysis model predicting 
participation in an advanced mathematics course sequence in high school. 


Table D-4. Results of Main Impact Model Predicting Likelihood of AR Students Taking 
Advanced Course Sequence in High School 


Variable 

Logit coefficient 

Standard 

error 

/?- value 

Odds ratio 

95% confidence 
interval 

Intercept 

-1.13 

0.29 

<0.001 

0.32 

(0.18, 0.57) 

School covariate 

Condition 

1.10 

0.39 

0.007 

2.99 

(1.38, 6.47) 

Medium-size school in Maine 

0.33 

0.50 

0.512 

1.39 

(0.52, 3.74) 

Large-size school in Maine 

0.57 

0.75 

0.454 

1.76 

(0.39, 7.94) 

Small-size school in Vermont 

0.64 

0.68 

0.348 

1.90 

(0.49, 7.33) 

Medium-size school in Vermont 

-0.13 

0.85 

0.877 

0.88 

(0.16,4.80) 

Large-size school in Vennont 

0.28 

0.74 

0.708 

1.32 

(0.30, 5.79) 

Student covariate 

Female 

0.12 

0.24 

0.612 

0.89 

(0.56, 1.41) 

Receives special education services 

-1.90 

1.13 

0.095 

0.15 

(0.02, 1.40) 

Eligible for free or reduced-price lunch 

0.04 

0.30 

0.885 

1.04 

(0.58, 1.87) 

State mathematics score (standardized) 3 

1.11 

0.24 

<0.001 

3.05 

(1.92,4.84) 


Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced. 


a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

Source: Coursetaking data collected from high schools study participants attended in 2009/10. 


The study team conducted six sensitivity analyses that test the robustness of the observed 
estimates from the benchmark impact model. These analyses test models with different methods 
of accounting for students’ baseline mathematics achievement, a model with no covariates, a 
model based on observed (nonimputed) data only as well as a model that tests an alternative 
nesting structure (with students in treatment schools clustered within online teachers). The 
results of these sensitivity analyses of the algebra outcome are reported in appendix E (models 
E.7-E.12). 


Secondary Impact Analyses 

Analyses of the four secondary research questions estimate the possible side effects of the 
intervention on AR and N-AR students. 


Effects on AR Students’ General Mathematics Scores. The first secondary analysis examines 
the effects of the intervention on AR students’ general mathematics scores. Because AR students 
who have access to online Algebra I could have less exposure to general grade 8 mathematics 
content than they would have had in the absence of the course, the study team hypothesized that 
their scores on a general mathematics assessment could be negatively affected. As noted in 
chapter, 3, the online Algebra I teachers taught significantly more algebra content and 
significantly less other mathematical content than teachers in control schools taught their high 
achievers. 
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For this analysis, researchers calculated the intent-to-treat estimate of the effect of treatment 
using the general mathematics achievement score (Promise Assessment) at the end of grade 8 as 
the outcome. They employed the same two-level model presented above for the general 
mathematics achievement score: 


GenMathjj = ( 3 o, + P i, ( StateMathy - StateMath..) + P2j ( SpEdjj - SpEd..) + ^y(FRLy - 

FRL.) + p ^(Femaley - Female..) + e,y. (D.3a) 


Pq/ = Yoo + Yoi TRTj + Y02 ( MaineMEDj - MaineMED .) + 703 ( MaineLGj - 

MaineLG.) + Y04 ( VermontSMj - VermontSM.) + Jos ( VermontMEDj - 


VermontMED.) + yo6 ( Vermont LGj - VermontLG.) + Uq,. (D.3b) 

Pi/ = Yio. (D.3c) 

Py = y 20 . (D.3d) 

P3j = Y3 o. (D.3e) 

p4/ = Y40. (D.3f) 


Table D-5 displays the results of the impact analysis model predicting achievement in general 
mathematics at the end of grade 8. There were no significant differences in general mathematics 
scores for AR students in treatment and control schools. 
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Table D-5. Results of Impact Model Predicting AR Students’ General Mathematics Posttest 
Scores 


Variable 

Coefficient 

Standard error 

/;- value 

Effect size 

Intercept 

357.82 

1.99 

<0.001 


School covariate 

Condition 

3.60 

2.80 

0.204 

0.14 

Medium-size school in Maine 

8.87 

3.46 

0.013 


Large-size school in Maine 

7.60 

4.95 

0.130 


Small-size school in Vermont 

12.27 

5.58 

0.032 


Medium-size school in Vermont 

11.22 

6.32 

0.080 


Large-size school in Vermont 

17.27 

5.64 

0.004 


Student covariate 

Female 

-6.01 

1.92 

0.002 


Receives special education services 
Eligible for free or reduced-price 

-7.65 

6.46 

0.241 


lunch 

0.02 

2.20 

0.993 


State mathematics score 
(standardized) 3 

18.73 

2.48 

<0.001 



Variance 

Chi-squared 
(degrees of freedom) 

/;- value 


Residual 

363.30 




Level 2 (school) 

41.03 

102.07 (61) 

0.001 


Total variance 

404.33 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (25.22). 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: General mathematics scores on study-administered Promise Assessment posttest. 


Effects on N-AR Students’ Algebra Scores, General Mathematics Scores, and Planned 
Coursetaking. The central question of this study concerns the effects of offering online Algebra 
I in a school that does not otherwise offer Algebra I to grade 8 students. The indirect effects of 
the intervention on N-AR students are of secondary importance. If there are direct effects on AR 
students, however, the secondary impact analyses are important for detennining whether the 
intervention is a good investment for schools interested in broadening access to algebra. 

The approach to estimating the impact of online Algebra I on N-AR student outcomes mirrors 
the analytic strategy for estimating the impacts of the intervention on AR students. All models 
used the N-AR student sample and estimated the effects of school-level treatment status on 
student-level outcomes. Specifically, equations D.la-D.lf were used to estimate N-AR students’ 
algebra achievement at the end of grade 8, equations D.2a-D.2f were used to estimate N-AR 
students’ planned grade 9 enrollments, and equations D.3a- D.3f were used to estimate N-AR 
students’ general mathematics achievement at the end of grade 8. Results of the secondary 
analysis models are presented in tables D-6 through D-9. These analyses reveal no significant 
differences between N-AR students in treatment schools and control schools on the algebra 
posttest, general mathematics posttest, or planned grade 9 enrollments. 

All analyses on the N-AR sample were conducted using 10 multiply imputed datasets; estimates 
presented in this appendix are averages based on the 10 different datasets. For more information 
on the multiple imputation of missing data, see appendix F. 
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Table D-6. Results of Impact Model Predicting N-AR Students’ Algebra Posttest Scores 


Variable 

Coefficient 

Standard error 

/?-value 

Effect size 

Intercept 

429.80 

1.01 

<0.001 


School covariate 

Condition 

0.96 

1.25 

0.443 

0.06 

Medium-size school in Maine 

-4.85 

1.91 

0.014 


Large-size school in Maine 

-5.81 

2.41 

0.019 


Small-size school in Vermont 

-7.93 

2.95 

0.010 


Medium-size school in Vermont 

-5.76 

3.01 

0.060 


Large-size school in Vermont 

-5.09 

2.33 

0.033 


Student covariate 

Female 

1.79 

0.90 

0.047 


Receives special education services 
Eligible for free or reduced-price 

-3.02 

1.51 

0.051 


lunch 

State mathematics score 

-1.34 

0.86 

0.119 


(standardized) 3 

3.87 

0.65 

<0.001 



Variance 

Chi-squared 
(degrees of freedom) 

/7-value 


Residual 

213.85 




Level 2 (school) 

8.23 

106.12(57) 

<0.001 


Total variance 

222.08 





Note : Sample included 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control schools); 4 
control schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was 
calculated using a pooled standard deviation of the outcome for N-AR students in treatment and control schools that 
incorporates both within and between imputation variance (16.94). 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source : Algebra scores on study-administered Promise Assessment posttest. 
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Table D-7. Results of Impact Model Predicting N-AR Students’ General Mathematics 
Posttest Scores 


Variable 

Coefficient 

Standard error 

/?-value 

Effect size 

Intercept 

324.21 

1.96 

<0.001 


School covariate 

Condition 

0.65 

2.41 

0.789 

0.02 

Medium-size school in Maine 

1.29 

3.38 

0.704 


Large-size school in Maine 

-4.97 

4.42 

0.266 


Small-size school in Vermont 

2.22 

5.21 

0.671 


Medium-size school in Vermont 

2.24 

5.66 

0.694 


Large-size school in Vermont 

8.90 

4.87 

0.072 


Student covariate 

Female 

-2.59 

1.41 

0.066 


Receives special education services 
Eligible for free or reduced-price 

-11.46 

2.27 

<0.001 


lunch 

-3.43 

1.45 

0.018 


State mathematics score 
(standardized) 3 

13.78 

0.98 

<0.001 



Variance 

Chi-squared (degrees 
of freedom) 

/;-value 


Residual 

583.17 




Level 2 (school) 

39.54 

133.04 (57) 

<0.001 


Total variance 

622.71 





Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was calculated 
using a pooled standard deviation of the outcome for N-AR students in treatment and control schools that incorporates both 
within and between imputation variance (31.44).. 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: General mathematics scores on study-administered Promise Assessment posttest. 


Table D-8 presents the descriptive statistics that mirror the secondary impact analyses for N-AR 
students in table 4-5 of the main text. Multiply imputed percentages presented in table D-8 are 
averaged across the 10 imputed datasets. The results indicate that among N-AR students, 76% 
students in treatment schools and 66% in control schools planned to enroll in an intennediate 
mathematics course in grade 9, a difference that was not statistically significant. 


Table D-8. Percentage of Grade 8 N-AR Students Who Planned to Enroll in Intermediate 
Mathematics Course in Grade 9 


Planned Grade 9 course 

Students in 
treatment schools 

Students in control 
schools 

All N-AR 

students 

Intermediate 

76.0 

66.2 

71.2 

(course equivalent to or above Algebra I) 

Not intermediate 

24.0 

33.8 

28.8 

(course below Algebra 1) 


Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); control 
schools had N-AR students. Estimates were averaged across 10 multiply imputed datasets. Twenty-three N-AR students 
planned to enroll in a grade 9 course above Algebra 1, including 13 students (2%) from control schools and 10 students (1%) 
from treatment schools 

Source: Planned courses indicated by study students at end of Grade 8. 
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Table D-9 presents the complete results of the impact model presented in table 4-5, a model 
predicting N-AR students’ plans to follow an intermediate mathematics course sequence in 
grade 9. 


Table D-9. Results of Impact Model Predicting N-AR Students’ Planned Grade 9 
Coursetaking 


Variable 

Logit coefficient 

Standard 

error 

/?-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

1.03 

0.35 

0.005 

2.80 

(0.38, 5.67) 

School covariate 

Condition 

0.78 

0.46 

0.099 

2.18 

(0.86, 5.51) 

Medium-size school in Maine 

1.37 

0.56 

0.018 

3.95 

(1.28, 12.16) 

Large-size school in Maine 

0.72 

0.87 

0.416 

2.05 

(0.36, 11.73) 

Small-size school in Vermont 

-0.46 

0.78 

0.559 

0.63 

(0.13,3.02) 

Medium-size school in Vermont 

-1.61 

0.92 

0.087 

0.20 

(0.03, 1.27) 

Large-size school in Vermont 

0.22 

0.84 

0.799 

1.24 

(0.23, 6.69) 

Student covariate 

Female 

0.08 

0.18 

0.637 

0.92 

(0.65, 1.31) 

Receives special education services 

-1.35 

0.24 

<0.001 

0.26 

(0.16,0.42) 

Eligible for free or reduced-price lunch 

-0.59 

0.16 

0.001 

0.56 

(0.40, 0.77) 

State mathematics score (standardized) 11 

1.54 

0.17 

<0.001 

4.66 

(3.32, 6.54) 


Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. Coursetaking sequences were 
coded as advanced or not advanced. 

a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Planned courses indicated by study students at end of Grade 8. 


Ten sensitivity analyses were conducted to test the robustness of the findings for N-AR students’ 
algebra achievement and planned grade 9 coursetaking. For each outcome, these sensitivity 
analyses tested three models with different methods of accounting for students’ baseline 
mathematics achievement, a model with no covariates, and a model based on observed 
(nonimputed) data only. The results of these sensitivity analyses are reported in appendix E (see 
models E. 13-E. 17 for the algebra outcome and models E. 18-E.22 for the grade 9 planned 
coursetaking outcome). 

Exploratory Analyses 

The two exploratory research questions were analyzed from 10 multiply imputed datasets. The 
model estimates presented are averages based on the 10 datasets. For more information on the 
multiple imputation of missing data, see appendix F. 

Exploratory Research Question 1 

To estimate the impact of access to online Algebra I on AR students’ planned coursetaking, the 
study team used the same analytic strategy used to estimate the impact of the intervention on AR 
students’ high school coursetaking. Using the AR student sample, researchers estimated the 
effect of school-level treatment status on students’ planned grade 9 enrollments, collected at the 
end of grade 8, with the models outlined in equations D.2a-D.2f. 
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Descriptive analyses showed that among AR students, 58% in treatment schools and just 24% in 
control schools initially registered for an advanced grade 9 course (table D-10). 55 

Table D-10. Percentage of Grade 8 AR Students Who Planned to Enroll in Advanced 
Mathematics Course in Grade 9 



Students in 

Students in 



treatment 

control 

All AR 

Planned course 

schools 

schools 

students 

Algebra I or below 

42.0 

75.7 

59.0 

Above Algebra I 

58.0 

24.3 

41.0 


Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. 

Source: Planned courses indicated by study students at end of Grade 8. 


Table D-l 1 displays the complete results of the model predicting AR students’ plans to follow an 
advanced mathematics course sequence in grade 9. 


Table D-ll. Results of Impact Model Predicting AR Students’ Planned Grade 9 
Coursetaking 


Variable 

Logit coefficient 

Standard 

error 

/?- value 

Odds ratio 

95% confidence 
interval 

Intercept 

-1.54 

0.46 

0.002 

0.21 

(0.09, 0.54) 

School covariate 

Condition 

1.78 

0.61 

0.005 

5.96 

(1.76,20.19) 

Medium-size school in Maine 

0.13 

0.72 

0.860 

1.14 

(0.27, 4.83) 

Large-size school in Maine 

-0.26 

1.19 

0.825 

0.77 

(0.07, 8.23) 

Small-size school in Vermont 

1.21 

0.99 

0.228 

3.35 

(0.46, 24.22) 

Medium-size school in Vennont 

-0.62 

1.32 

0.640 

0.54 

(0.04, 7.54) 

Large-size school in Vermont 

-0.51 

1.19 

0.670 

0.60 

(0.06, 6.45) 

Student covariate 

Female 

0.17 

0.28 

0.533 

0.84 

(0.49, 1.45) 

Receives special education services 

-0.45 

0.98 

0.649 

0.64 

(0.09, 4.42) 

Eligible for free or reduced-price lunch 

-0.35 

0.32 

0.282 

0.71 

(0.38, 1.33) 

State mathematics score (standardized) 3 

1.07 

0.30 

0.001 

2.91 

(1.62,5.22) 


Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced. 


a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

Source: Planned courses indicated by study students at end of Grade 8. 


These results are consistent with the main results on AR students’ high school coursetaking, and 
together they show that students with access to online Algebra I in grade 8 were more likely to 
both initially enroll in an advanced grade 9 course at the end of grade 8 (planned grade 9 
coursetaking) and to continue to follow an advanced course sequence as of the end of grade 9 
(actual coursetaking in grade 9 and planned coursetaking in grade 10). 


55 The percentages of students who completed an advanced course in grade 9 were 59% for treatment 
group students and 31% for control group students (see table D-3). 
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Exploratory Research Question 2 

To estimate the impact of online Algebra I on AR students’ likelihood of doubling up on 
mathematics courses in grade or grade 10, researchers used the same impact model (outlined in 
equations 2a-2f) with one exception. The model for doubling up on mathematics courses did not 
include students’ special education status as a covariate at level 1 because no students with 
special education status doubled up. Results for exploratory question 2 are presented in Tables 
D-12 and D-13. 

Table D-12 presents the descriptive results for doubling up by condition. As shown below, 14% 
of AR students from treatment middle schools doubled-up on mathematics courses in grade 9 or 
10, compared with 29% of AR students from control middle schools. 

Table D-12. Percentage of AR Students Who Double Up on Mathematics Courses in Grade 
9 or 10 


Coursetaking pattern 

Students from 
treatment schools 

Students from 
control schools 

All AR students 

Did not double up in grade 9 or 10 

85.6 

70.8 

78.2 

Doubled up in grade 9 or 10 

14.4 

29.2 

21.8 


Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Doubling up was coded as AR students who took two or more mathematics 
courses in grade 9 or 10. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Table D-13 displays the complete results of the model predicting AR students’ likelihood of 
doubling up on mathematics courses in grade 9 or 10. 

Table D-13. Results of Impact Model Predicting AR Students’ Likelihood of Doubling Up 
on Full-Year Mathematics Courses in Grade 9 or 10 


Variable 

Logit coefficient 

Standard 

error 

/;- value 

Odds ratio 

95% confidence 
interval 

Intercept 

-1.27 

0.33 

<0.001 

0.28 

(0.15,0.54) 

School covariate 

Condition 

-1.07 

0.49 

0.033 

0.34 

(0.13,0.92) 

Medium-size school in Maine 

-0.24 

0.61 

0.700 

0.79 

(0.23, 2.69) 

Large-size school in Maine 

-0.39 

0.96 

0.685 

0.68 

(0.10,4.58) 

Small-size school in Vermont 

-0.33 

0.88 

0.706 

0.72 

(0.12, 4.16) 

Medium-size school in Vennont 

1.18 

0.96 

0.225 

3.25 

(0.48, 22.18) 

Large-size school in Vennont 

1.64 

0.82 

0.050 

5.14 

(1.00, 26.46) 

Student covariate 

Female 

-0.18 

0.30 

0.536 

1.20 

(0.67,2.15) 

Eligible for free or reduced-price lunch 

-0.10 

0.37 

0.777 

0.90 

(0.44, 1.85) 

State mathematics score (standardized) 

0.35 

0.26 

0.177 

1.42 

(0.85,2.35) 


Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Doubling up was coded as 1 for AR students who took 3 or more mathematics 
courses in grade 9 or 10 and 0 for students who took 2 or fewer mathematics courses. 

a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 
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These results suggest that AR students in controls schools who do not have access to Algebra I in 
grade 8 were more likely to double up on mathematics courses in grade 9 or 10 than students in 
treatment schools, possibly to get “on track” for an advanced course sequence. 
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APPENDIX E: SENSITIVITY ANALYSES 


This appendix presents the models used to conduct sensitivity analyses. As in appendix D, this 
appendix presents coefficients and standard errors for all of the variables in each model, 
including student- and school-level covariates. The results for coursetaking sensitivity analyses 
also include corresponding odds ratios and 95 percent confidence intervals for each predictor. 

Impact of Intervention on AR Students' Algebra Scores 

To determine the robustness of the impact estimates based on the benchmark impact model for 
AR students’ algebra posttest scores, the study team conducted the following sensitivity 
analyses: 

• Model E. 1 includes prior state mathematics assessment scores and study-administered 
pretest scores at Level 1 (grand mean centered). 

• Model E.2 replaces students’ prior state mathematics assessment scores with their 
study-administered pretest scores at Level 1 (grand mean centered). 

• Model E.3 does not adjust for baseline mathematics achievement by excluding 
students’ prior state mathematics assessment scores at Level 1. 

• Model E.4 excludes all covariates at Level 1 and Level 2. 

• Model E.5 tests the benchmark impact model on observed (nonimputed) data only. 

• Model E.6 combines the 34 schools that share an online teacher into 8 “pseudo- 
schools” for treatment students and uses the standard school for all control students. 

• Model E.7 includes a covariate that captures time spent on the algebra posttest at 
Level 1 (grand-mean centered). 

Sensitivity Analyses Using Different Methods of Adjusting for Baseline Mathematics 
Achievement 

The first three sensitivity analyses examined the extent to which the estimate of the impact of the 
intervention is robust to different ways of controlling for baseline mathematics achievement. In 
three models researchers tested whether the treatment effect is robust when (a) the study- 
administered Promise Assessment pretest scores is added to state mathematics scores, (b) 
students’ baseline state mathematics scores are replaced with their pretest scores, and (c) both 
measures of prior mathematics achievement are excluded. As detailed in chapter 2, the study- 
administered pretest was endogenous to the study — that is, it was administered after random 
assignment and after implementation of the intervention had begun in treatment schools. 
Additionally, students in treatment and control schools differed significantly in their pretest 
scores at baseline. Scores on the state mathematics assessments from the previous year, however, 
were exogenous to the study and could not have been affected by the presence of the study or the 
intervention. Students in treatment and control schools did not differ in their prior state 
mathematics scores at baseline. 


El 


Sensitivity Analyses for AR Students ’ Algebra Scores 

All of the sensitivity analyses are consistent with the results of the benchmark impact analysis 
model, where the estimated impact was 5.53 (p = 0.001, effect size = 0.39) (table E-l). In each 
analysis, the effect of condition on AR students’ algebra scores at the end of grade 8 was 
statistically significant (using an alpha value of 0.025 that accounts for multiple comparisons), 
with AR students in treatment schools outperforming their counterparts in control schools. In 
model E.2, where student’s state mathematics assessment scores are replaced with their study- 
administered pretest scores, the treatment effect is attenuated: the effect size (0.29) is 26% 
smaller than the effect size reported for the impact model (0.39). The significant difference by 
condition on the pretest may explain this attenuation. 


Table E-l. Summary of Sensitivity Analyses Testing Robustness of Effect of Condition 
on AR Students’ Algebra Posttest Scores 


Model 

Estimated impact 
(standard error) 

/j-value 

Effect size 

E.l 

includes prior state mathematics and study-administered 
pretest scores as covariates at Level 1 (grand-mean 
centered) 

5.05 

(1.52) 

0.002 

0.37 

E.2 

replaces state mathematics with study-administered 
pretest scores as covariates at Level 1 

4.01 

(1.47) 

0.009 

0.29 

E.3 

excludes state mathematics and study-administered 
pretest scores as covariates at Level 1 

5.35 

(1.83) 

0.005 

0.39 

E.4 

excludes all covariates at Level 1 and Level 2 

5.17 

(1.82) 

0.007 

0.38 

E.5 

tests benchmark impact model using observed 
(nonimputed) data only 

5.59 

(1.51) 

0.001 

0.45 a 

E.6 

clusters treatment students who share an online teacher 
into “pseudo-schools” and control students by their 
standard school 

5.55 

(1.80) 

0.004 

0.41 

E.7 

includes a covariate that captures time spent on the 
algebra posttest (grand-mean centered) 

4.79 

(1.47) 

0.002 

0.35 


Note : Sample for all models except E.5 includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 
control); estimates are averaged across 10 multiply imputed datasets. Model E.5 includes 68 schools (35 treatment, 33 control) 
and 430 AR students); 10 students were missing data on the outcome or 1 covariate (estimates are based on observed data 
only). The sample size of 430 is different from the analytic sample of 440 because this model uses only observed (nonimputed) 
data. Model E.6 excludes state by size blocking covariates at Level 2, given that they do not apply to online teachers. Effect 
sizes were calculated using a pooled standard deviation of the outcome for AR students in treatment and control schools that 
incorporates both within and between imputation variance (13.78). 

a. The Eledges’ g effect size was calculated using a pooled standard deviation of the outcome for AR students in treatment and 
control schools (13.77). 

Source: Algebra scores on study-administered Promise Assessment posttests. 
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Model E.l 


Model E. 1 is identical to the model estimated for the benchmark analysis, except that students’ 
study-administered pretest scores is added as an adjustment for baseline mathematics 
achievement at Level 1 . As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


Algebra^- = Pq/ + Pi/ ( StateMathj - StateMath..) + P2j ( Pretest] - Pretest..) + P3j (SpEd,, - 
SpEd . .) + P 4 y ( FRLjj - FRL . .) + p 5y ( Female y - Female . .) + s y. (E. 1 a) 


Pq/ = Yoo + Yoi PR Tj + Y02 ( MaineMEDj - MaineMED.) + 703 ( MaineLGj - 

MaineLG.) + Y04 ( VermontSMj - VermontSM.) + y 05 ( VermontMEDj - 


VermontMED .) + yo6 ( Vermont LGj - VermontLG.) + uq/. (E. lb) 

Py = Yio- (E.lc) 

p 2 , = y 20 . (E.ld) 

p3/ = y.30- (E.le) 

p 4 , = y 4 o. (E.if) 

p 5 / = y 5 o. (E.ig) 


Table E-2. Results of Sensitivity Model 1: Adding Study-Administered Pretest Scores at 
Level 1 (Grand Mean Centered) 


Variable 

Coefficient 

Standard error 

/?- value 

Effect size 

Intercept 

441.93 

1.09 

<0.001 


School covariate 

Condition 

5.05 

1.52 

0.002 

0.37 

Medium-size school in Maine 

1.30 

1.93 

0.501 


Large-size school in Maine 

2.72 

2.66 

0.311 


Small-size school in Vermont 

4.60 

2.99 

0.129 


Medium-size school in Vennont 

2.24 

3.42 

0.515 


Large-size school in Vermont 

2.54 

2.94 

0.391 


Student covariate 

Female 

0.49 

1.15 

0.669 


Receives special education services 

-10.38 

3.45 

0.003 


Eligible for free or reduced-price 
lunch 

2.08 

1.29 

0.107 


Pretest 

0.07 

0.04 

0.119 


State mathematics score 
(standardized) 3 

8.58 

1.56 

<0.001 



Variance 

Chi-squared 
(degrees of freedom) 

/;- value 


Residual 

127.15 




Level 2 (school) 

9.46 

83.18(61) 

0.031 


Total variance 

136.61 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 
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Model E.2 


Model E.2 is identical to the model estimated for the benchmark analysis, except that students’ 
state mathematics assessment scores are replaced with their scores on the study-administered 
pretest at Level 1 . As was the case in the benchmark analysis, all covariates except the treatment 
indicator were centered on the grand mean. 


Algebra^- = Pq/ + Pi/ ( Pretest y - Pretest..) + P2j ( SpEdj - SpEd..) + p3/ ( FRLjj - FRL..) + p4/ 
(. Femaleij - Female..) + s,y. (E.2a) 

Pq/ = Yoo + Yoi TRTj + Y 02 ( MaineMEDj - MaineMED .) + Y 03 (Mai net G, - 
MaineLG.) + Y 04 ( VermontSMj - VermontSM.) + Y 05 (VermontMEDj - 


VermontMED.) + Y 06 ( VermontLGj - VermontLG.) + Uq,. (E.2b) 

Pi/ = Yio. (E.2c) 

P 2/ = Y 20 . (E.2d) 

P 3 / = Y30. (E.2e) 

p 4/ = Y40. (E.2f) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on algebra scores is robust to replacing students’ state mathematics scores 
with their pretest scores as a different (although endogenous) measure of baseline 
mathematics achievement (table E-3). 

Table E-3. Results of Sensitivity Model 2: Replacing Prior State Mathematics Assessment 
Scores with Study-Administered Pretest Scores at Level 1 


Variable 

Coefficient 

Standard error 

p-value 

Effect size 

Intercept 

442.56 

1.09 

<0.001 


School covariate 

Condition 

4.01 

1.47 

0.009 

0.29 

Medium-size school in Maine 

1.17 

1.99 

0.557 


Large-size school in Maine 

3.09 

2.67 

0.252 


Small-size school in Vermont 

4.80 

2.72 

0.082 


Medium-size school in Vennont 

0.25 

3.34 

0.942 


Large-size school in Vermont 

1.52 

2.54 

0.550 


Student covariate 

Female 

-1.49 

1.21 

0.220 


Receives special education services 

-9.70 

3.29 

0.004 


Eligible for free or reduced-price 
lunch 

1.51 

1.32 

0.256 


Pretest 

0.23 

0.03 

<0.001 



Variance 

Chi-squared (degrees 
of freedom) 

p-value 


Residual 

147.57 




Level 2 (school) 

7.65 

77.71 (61) 

0.073 


Total variance 

155.22 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
Source: Algebra scores on study-administered Promise Assessment posttest. 
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Model E.3 


Model E .3 is identical to the model estimated for the benchmark analysis, except that the study 
team excluded students’ state mathematics assessment scores at Level 1 and controlled only for 
student demographic characteristics (gender, eligibility for free or reduced priced lunch, special 
education status) at Level 1 and state by size blocking variables (used for random assignment) at 
Level 2 . As was the case in the benchmark analysis, all covariates except the treatment indicator 
were centered on the grand mean. 


Algebra^- = | 3 q/ + ( 3 ij ( SpEdy - SpEd..) + (fi/ ( FRLy - FRL..) + (A,, (Femaley - Female..) + 
s y, (E .3 a) 

po/ = Yoo + Yoi TRTj + 702 ( MaineMEDj - MaineMED .) + 703 ( MaineLGj - 

MaineLG .) + yo4 ( VermontSM/ - VermontSM.) + jo5 ( VermontMEDj - 


VermontMED.) + yo6 ( VermontLG ) - VermontLG.) + Uq/. (E. 3 b) 

Pi/ = Yio. (E. 3 c) 

p2, = Y20. (E. 3 d) 

p3/ = Y 30 . (E. 3 e) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on algebra scores is robust when not adjusting for students’ mathematics 
achievement at baseline (that is, excluding state mathematics and pretest scores) (table E- 4 ). 

Table E-4. Results of Sensitivity Model 3: Excluding Students’ Prior State Mathematics 
Assessment Scores at Level 1 


Variable 

Coefficient 

Standard error 

/;- value 

Effect size 

Intercept 

441.83 

1.34 

<0.001 


School covariate 

Condition 

5.35 

1.83 

0.005 

0.39 

Medium-size school in Maine 

1.88 

2.39 

0.436 


Large-size school in Maine 

2.08 

3.47 

0.551 


Small-size school in Vermont 

5.08 

3.27 

0.125 


Medium-size school in Vennont 

-0.41 

4.05 

0.921 


Large-size school in Vermont 

3.79 

3.23 

0.246 


Student covariate 

Female 

-0.06 

1.27 

0.963 


Receives special education services 

-13.34 

3.50 

<0.001 


Eligible for free or reduced-price 
lunch 

0.75 

1.41 

0.594 



Variance 

Chi-squared 
(degrees of freedom) 

/>- value 


Residual 

162.87 




Level 2 (school) 

22.09 

112.36 (61) 

<0.001 


Total variance 

184.96 




Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 


for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
Source: Algebra scores on study-administered Promise Assessment posttest. 
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Model E.4 


To examine the relationship between the effect of the intervention on algebra scores and the 
covariates specified in the benchmark impact model, model E.4 tests the treatment effect while 
excluding students’ state mathematics assessment scores and their demographic characteristics 
(gender, eligibility for free or reduced priced lunch, special education status) at Level 1 and state 
by size blocking variables (used for random assignment) at Level 2. 

Algebraij = p 0/ + % (E.4a) 

Po/ = Too + Yoi TRTj + uq/. (E.4b) 


A statistically significant and positive coefficient y 0 i for treatment indicates that the effect of 
the online course on algebra scores is robust when not adjusting for student-level variability 
in baseline mathematics achievement and demographic characteristics and school-level 
variability across size and state (table E-5). 


Table E-5. Results of Sensitivity Model 4: Excluding All Covariates at Level 1 or 2 


Variable 

Coefficient 

Standard error 

/7-value 

Effect size 

Intercept 

442.22 

1.30 

<0.001 


School covariate 

Condition 

5.17 

1.82 

0.007 

0.38 


Variance 

Chi-squared (degrees 
of freedom) 

p-value 


Residual 

166.33 




Level 2 (school) 

23.25 

125.08 (66) 

<0.001 


Total variance 

189.58 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
Source: Algebra scores on study-administered Promise Assessment posttest. 
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Model E.5 


To examine whether the effect of the intervention on algebra posttest scores is sensitive to the 
missing data approach (multiple imputation, described in appendix F), model E .5 tests the 
benchmark impact model using observed (nonimputed) data only, excluding cases that were 
missing data for the algebra posttest or a covariate in the analytic model ( 2 . 3 % of all cases). The 
loss of data reduces the size of the available sample and associated statistical power and may 
introduce bias into the parameter estimate. 56 However, the impact models include covariates 
found to be related to missing values, such as gender, eligibility for free or reduced-price lunch, 
state mathematics assessment scores, and school-level state and size blocking covariates. 
Inclusion of these covariates may help reduce bias caused by missing data. 57 As was the case in 
the benchmark analysis, all covariates except the treatment indicator were centered on the grand 
mean. 


Algebra^- = | 3 q/ + P i 7 ( StateMathy - StateMath..) + (T, ( SpEdy - SpEd..) + (T v (FRLjj - 

FRL..) + fi+j ( Female^ - Female. .) + £,y. (E. 5 a) 

Pq; = Too + Yoi TRTj + 702 ( MaineMEDj - MaineMED .) + 703 ( MaineLGj - 

MaineLG.) + yo4 ( VermontSMj - VermontSM.) + yos ( VermontMEDj - 


VermontMED.) + yo6 ( Vermont LGj - VermontLG.) + Uq,. (E. 5 b) 

Pi/ = Yio. (E. 5 c) 

p2/ = Y20. (E. 5 d) 

p3/ = Y3 o. (E. 5 e) 

p4, = y 40 . (E. 5 f) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on algebra scores is robust to the method of handling missing data 
(multiply imputed versus observed only) (table E-6). 


56 This loss of data could lead to larger standard errors, wider confidence intervals, and loss of power in testing 
hypotheses. 

57 When the listwise approach is used, it is usually recommended to include in the regression used to 
estimate impact covariates that may influence both the outcome and the probability of having missing 
data on outcomes (Puma et al. 2009). 
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Table E-6. Results of Sensitivity Model 5: Including Observed (Nonimputed) Data 
Only 


Variable 

Coefficient 

Standard error 

/7-value 

Effect size 

Intercept 

441.76 

1.12 

<0.001 


School covariate 

Condition 

5.59 

1.51 

0.001 

0.41 a 

Medium-size school in Maine 

1.49 

2.01 

0.461 


Large-size school in Maine 

2.73 

2.85 

0.341 


Small-size school in Vermont 

4.49 

2.77 

0.110 


Medium-size school in Vennont 

-0.14 

3.39 

0.968 


Large-size school in Vermont 

1.60 

2.66 

0.548 


Student covariate 

Female 

-0.25 

1.14 

0.827 


Receives special education services 

-9.56 

3.15 

0.003 


Eligible for free or reduced-price 
lunch 

2.37 

1.27 

0.062 


State mathematics score 
(standardized) 13 

10.12 

0.95 

<0.001 



Variance 

Chi-squared (degrees 
of freedom) 

p-value 


Residual 

131.15 




Level 2 (school) 

12.23 

90.70 (61) 

0.008 


Total variance 

143.38 





AR is algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 430 AR students (215 treatment, 215 control); 10 students were 
missing data on the outcome or 1 covariate. The sample size of 430 is different from the analytic sample of 440 because this table 
represents a sensitivity analysis using only observed (not imputed) data. 

a. The Hedges’ g effect size was calculated using a pooled standard deviation of the outcome for AR students in treatment and 
control schools (13.77). 

b. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source : Algebra scores on study-administered Promise Assessment posttest. 


Model E.6 

The outcomes for treatment students in schools that share a common online teacher may be 
affected in similar ways by specific qualities and background of the online teacher (for example, 
the teacher’s knowledge or experience teaching online). Consequently, students in treatment 
schools that share an online teacher are not completely independent. To examine whether the 
observed treatment effect on algebra scores in the benchmark analysis is robust to the clustering 
of students within online teachers, researchers tested a model similar to the model estimated for 
the benchmark analysis, with two exceptions. First, treatment students were clustered at Level 2 
within the eight online Algebra I teachers, and control students were clustered within their 
standard schools. Second, because school-level covariates (school state and size) do not apply to 
online teachers, only the treatment indicator was included in this model at Level 2. As was the 
case in the benchmark analysis, all covariates except the treatment indicator were centered on the 
grand mean. 


Algebra^- = |3q/ + |) i, ( StateMathj - StateMath..) + Psj ( SpEdj - SpEd..) + [L v ( FRLjj - 

FRL..) + p 4y - (Female,! - Female. .) + s,y. (E.6a) 

Poy = Too + Yoi TRTj + up/. (E.6b) 
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Ply = Yio. 

(E.6c) 

p 2 y = Y 20 . 

(E.6d) 

-TO 

■is 

II 

u? 

0 

(E.6e) 

P4 i = Y40. 

(E.6f) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on algebra scores is robust to the unit of clustering at Level 2 for treatment 
students by using online teacher instead of school (table E-7). 


Table E-7. Results of Sensitivity Model 6: Using Pseudo-School Level 2 Clusters in 
Estimation of AR Students’ Algebra Scores 


Variable 

Coefficient 

Standard error 

/>-value 

Effect size 

Intercept 

441.95 

1.05 

<0.001 


School covariate 

Condition 

5.55 

1.80 

0.004 

0.40 

Student covariate 

Female 

-0.17 

1.13 

0.884 


Receives special education services 

-11.04 

3.60 

0.003 


Eligible for free or reduced-price 
lunch 

1.76 

1.29 

0.171 


State mathematics score 
(standardized) 3 

9.92 

1.11 

<0.001 



Variance 

Chi-squared 
(degrees of freedom) 

/?- value 


Residual 

130.01 




Level 2 (school) 

9.44 

65.49 (45) 

0.024 


Total variance 

139.45 





AR is algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Model E.7 

Model E.7 is identical to the model estimated for the benchmark analysis, except that students’ 
time spent on the algebra posttest is added as a covariate at Level 1 . As noted in Chapter 2 and 
Appendix B, there was an observed difference by condition in the amount of time spent by AR 
students on the computer-adaptive mathematics pretest, where students in treatment schools 
spent significantly more time than students in control schools. The difference in time spent on 
the algebra posttest was not significantly different by condition but time spent on the test could 
nevertheless account for all or part of the observed treatment effect. This sensitivity analysis was 
conducted to examine this possibility. As was the case in the benchmark analysis, all covariates 
except the treatment indicator were centered on the grand mean. 
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Algebrcty = |3q/ + P i 7 ( StateMathy - StateMath..) + ^ 2 j ( SpEdy - SpEd..) + p 3 ; (FRLjj - 
FRL..) + p 4y - (Female^ - Female.) + p 5 ; (Time on Algebra Posttesty - Time on 
Algebra Posttest.) + s y. (E.7a) 

P Q/ = Yoo + Yoi TRTj + y 0 2 ( MaineMEDj - MaineMED ) + jo 3 ( MaineLGj - 

MaineLG) + jo 4 ( VermontSMj - VermontSM) + yos ( VermontMEDj - 


VermontMED) + yo6 ( Vermont LGj - VermontLG) + Uq,. (E.7b) 

Pi/ = Yio- (E.7c) 

p 2/ = Y 20 . (E.7d) 

P3/ = Y30- (E.7e) 

p4, = Y40. (E.7f) 

p5j = Y50. (E.7g) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on algebra scores detected in the main analysis is not explained by 
differences by condition in the amount of time spent on the algebra posttest by AR students 
(table E-8). 

Table E-8. Results of Sensitivity Model 7: Adding Time Spent on Algebra Posttest at 
Level 1 (Grand Mean Centered). 


Variable 

Coefficient 

Standard error 

/;-valuc 

Effect size 

Intercept 

441.86 

1.05 

<0.001 


School covariate 

Condition 

4.79 

1.47 

0.002 

0.35 

Medium-size school in Maine 

2.52 

1.87 

0.185 


Large-size school in Maine 

2.91 

2.59 

0.266 


Small-size school in Vermont 

3.90 

2.92 

0.187 


Medium-size school in Vennont 

4.53 

3.34 

0.179 


Large-size school in Vermont 

3.61 

2.85 

0.211 


Student covariate 

Female 

0.88 

1.11 

0.430 


Receives special education services 

-12.67 

3.32 

<0.001 


Eligible for free or reduced-price 
lunch 

2.58 

1.25 

0.039 


State mathematics score 
(standardized) 3 

8.74 

1.10 

<0.001 


Time spent on Algebra Posttest 

0.52 

0.10 

<0.001 



Variance 

Chi-squared 
(degrees of freedom) 

/;-value 


Residual 

119.22 




Level 2 (school) 

8.98 

85.07 (61) 

0.022 


Total variance 

128.20 





Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. The effect size was calculated using a pooled standard deviation of the outcome 
for AR students in treatment and control schools that incorporates both within and between imputation variance (13.78). 
b. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 
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Impact of Intervention on AR Students’ High School Coursetaking 

To determine the robustness of the impact estimates based on the benchmark impact model for 
high school coursetaking, researchers conducted the same analyses conducted for the algebra 
posttest: 

• Model E.8 includes prior state mathematics assessment scores and study-administered 
pretest scores at Level 1 (grand mean centered). 

• Model E.9 replaces students’ prior state mathematics assessment scores with their 
study-administered pretest scores at Level 1 (grand mean centered). 

• Model E. 10 does not adjust for baseline mathematics achievement by excluding 
students’ prior state mathematics assessment scores at Level 1. 

• Model E. 1 1 excludes all covariates at Level 1 and Level 2. 

• Model E. 12 tests the benchmark impact model on observed (nonimputed) data only. 

• Model E. 13 combines the 34 schools that share an online teacher into 8 “pseudo- 
schools” for treatment students and uses the standard school for all control students. 

All but one of these sensitivity analyses is consistent with the results of the benchmark impact 
analysis model, where the estimated logit coefficient was 1.10 (p = 0.007, odds ratio = 2.99) 
(table E-9). In model E.9, in which students’ state mathematics assessment scores are replaced 
with their study-administered pretest scores, the treatment effect became nonsignificant (p = 
0.037), using an alpha value of 0.025 that accounts for multiple comparisons. The significant 
difference by condition on the pretest may (at least in part) explain this change in significance. In 
each of the other sensitivity analyses, the effect of the intervention on AR students’ high school 
coursetaking was statistically significant, with AR students in treatment schools more likely to 
follow an advanced course sequence than their counterparts in control schools. 


Ell 


Table E-9. Summary of Sensitivity Analyses Testing the Robustness of the Effect of 
Condition on AR Students’ High School Coursetaking 


Model 

Logit coefficient 
(standard error) 

p-value 

Odds ratio 
(95% confidence 
interval) 

E.8 

0.97 

0.017 

2.63 

includes prior state mathematics and study- 
administered pretest scores as covariates at Level 1 

(0.39) 


(1.20, 5.78) 

E.9 

0.82 

0.037 

2.28 

replaces state mathematics with study-administered 
pretest scores as covariates at Level 1 

(0.39) 


(1.05,4.94) 

E.10 

0.96 

0.014 

2.60 

excludes state mathematics and study-administered 
pretest scores as covariates at Level 1 

(0.38) 


(1.22, 5.53) 

E.l 1 

0.89 

0.014 

2.44 

excludes all covariates at Level 1 and Level 2 

(0.35) 


(1.21,4.92) 

E.12 

1.17 

0.005 

3.22 

tests benchmark impact model using observed 
(nonimputed) data only 

(0.40) 


(1.46, 7.12) 

E.13 

1.08 

0.020 

2.94 

clusters treatment students who share an online teacher 
into “pseudo-schools” and control students by their 
standard school 

(0.45) 


(1.20, 7.22) 


AR is algebra ready. 


Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates for 
all models except E.12 are averaged across 10 multiply imputed datasets. Model E.l 1 includes 68 schools (35 treatment, 33 
control) and 424 AR students (210 treatment, 214 control); 16 students were missing data on the outcome or 1 covariate 
(estimates are based on observed data only). The sample size of 424 is different from the analytic sample of 440 because this 
model tests a sensitivity analysis using only observed (nonimputed) data. Coursetaking sequences were coded as advanced or 
not advanced. Model E.13 excludes state by size blocking covariates at Level 2, given that they do not apply to online teachers, 
Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Model E.8 

Model E.8 is identical to the model estimated for the benchmark analysis, except that students’ 
study-administered pretest scores are added as an additional adjustment for baseline mathematics 
achievement at Level 1 . As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


T| jj = po/+ Pi/ ( StateMathjj - StateMath..) + Pu ( Pretest y - Pretest..) + P3j ( SpEdjj - SpEd..) 
+ P 4j ( FRLjj - FRL..) + p 5 y (Femaley - Female.) (E.8a) 


where % = log (cp /y / 1 - cp ( y) 


Pq/ = Yoo + Yoi PR Tj + Y02 ( MaineMEDj - MaineMED .) + 703 ( MaineLGj - 
MaineLG.) + Y04 ( VermontSM ) - VermontSM .) + y 0 5 ( VermontMEDj - 
VermontMED.) + yo 6 ( VermontLGj - VermontLG .) + uq/ (E. 8 b) 
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Ply = YlO. 

(E.8c) 

P2y = 720 . 

(E.8d) 

-CO 

■is 

II 

0 

(E.8e) 

P4y = Y40. 

(E.8f) 

P5/ = Y50. 

(E.8g) 


A statistically significant and positive coefficient Yoi for treatment indicates that the effect of 
the online course on high school coursetaking is robust to adding the pretest as an additional 
covariate (table E-10). 

Table E-10. Results of Sensitivity Model 8: Adding Study-Administered Pretest Scores at 
Level 1 (Grand Mean Centered) 


Variable 

Logit coefficient 

Standard 

error 

/?-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

-1.06 

0.29 

0.001 

0.35 

(0.19, 0.62) 

School covariate 

Condition 

0.97 

0.39 

0.017 

2.63 

(1.20,5.78) 

Medium-size school in Maine 

0.26 

0.51 

0.613 

1.29 

(0.47, 3.56) 

Large-size school in Maine 

0.62 

0.77 

0.421 

1.86 

(0.40, 8.64) 

Small-size school in Vermont 

0.60 

0.68 

0.380 

1.83 

(0.47, 7.10) 

Medium-size school in Vennont 

-0.24 

0.87 

0.779 

0.78 

(0.14, 4.41) 

Large-size school in Vennont 

0.08 

0.74 

0.915 

1.08 

(0.25,4.75) 

Student covariate 

Female 

-0.24 

0.25 

0.335 

0.79 

(0.49, 1.28) 

Receives special education services 

-1.68 

1.12 

0.136 

0.19 

(0.02, 1.70) 

Eligible for free or reduced-price lunch 

-0.06 

0.30 

0.846 

1.06 

(0.59, 1.91) 

Pretest 

0.02 

0.01 

0.015 

1.02 

(1.00, 1.04) 

State mathematics score (standardized) 3 

0.72 

0.29 

0.014 

2.05 

(1.16,3.62) 


AR is algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced, 
a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Model E.9 

Model E.9 is identical to the model estimated for the benchmark analysis, except that students’ 
state mathematics assessment scores are replaced with their scores on the study-administered 
pretest at Level 1 . As was the case in the benchmark analysis, all covariates except the treatment 
indicator were centered on the grand mean. 


% = Po/+ Pi, {P retest ij - Pretest..) + p 2 j (SpEdy - SpEd..) + $y(FRLij - FRL.) + p 4/ 

( Femaleij - Female..) (E.9a) 

where % = log (cp ;> / 1 - cp (/ ) 
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Po/ = Yoo + Yoi TRTj + Y02 ( MaineMEDj - MaineMED .) + y 0 3 ( MaineLGj - 
MaineLG .) + Y 04 ( VermontSM ) - VermontSM.) + y 05 ( VermontMEDj - 


VermontMED.) + yo 6 ( VermontLGj - VermontLG.) + Uq/ (E.9b) 

Pi/ = Yio (E.9c) 

P2/ = Y2o (E.9d) 

P 3 / = Y 30 (E.9e) 

p4/ = Y4o. (E.9f) 


A statistically significant and positive coefficient yoi for treatment would indicate that the 
effect of the online course on high school coursetaking is robust to replacing students’ state 
mathematics scores with their pretest scores as a different (although endogenous) measure 
of baseline mathematics achievement (table E-l 1). 

Table E-ll. Results of Sensitivity Model 9: Replacing Students’ Prior State Mathematics 
Assessment Scores with Study-Administered Pretest Scores at Level 1 (Grand Mean 
Centered) 


Variable 

Logit coefficient 

Standard 

error 

/?-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

-0.95 

0.29 

0.002 

0.39 

(0.22, 0.69) 

School covariate 

Condition 

0.82 

0.39 

0.037 

2.28 

(1.05,4.94) 

Medium-size school in Maine 

0.21 

0.50 

0.682 

1.23 

(0.45, 3.35) 

Large-size school in Maine 

0.57 

0.77 

0.456 

1.78 

(0.39, 8.20) 

Small-size school in Vermont 

0.53 

0.67 

0.437 

1.69 

(0.44, 6.47) 

Medium-size school in Vennont 

-0.45 

0.86 

0.599 

0.64 

(0.12,3.52) 

Large-size school in Vennont 

-0.05 

0.71 

0.948 

0.95 

(0.23, 3.96) 

Student covariate 

Female 

-0.31 

0.24 

0.196 

0.731 

(0.46, 1.18) 

Receives special education services 

-1.58 

1.11 

0.157 

0.205 

(0.02, 1.84) 

Eligible for free or reduced-price lunch 

0.00 

0.29 

0.998 

0.999 

(0.57, 1.77) 

Pretest 

0.03 

0.01 

<0.001 

1.032 

(1.02, 1.05) 

AR is algebra ready. 


Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


As shown in Table E-ll, the results for Model 9 vary from the results of the benchmark analysis. 
The effect of the intervention is not statistically significant in Model 9 ;p = 0.037 (using a 
Bonferroni correction to yield a statistical significance level of p = 0.025). 

Model E.10 

Model E. 10 is identical to the model estimated for the benchmark analysis, except that 
researchers excluded students’ state mathematics assessment scores at Level 1 and controlled 
only for student demographic characteristics (gender, eligibility for free or reduced priced lunch, 
special education status) at Level 1 and state by size blocking variables (used for random 
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assignment) at Level 2 . As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


r\ y = po/+ Pi j(SpEdj - SpEd..) + $2j{FRLy -FRL..) + P y(Femaleij- Female..) (E.lOa) 
where % = log (cp /y / 1 - cp (/ ) 

Pq/ = Yoo + Yoi TRTj + Y02 ( MaineMEDj - MaineMED.) + 703 (MaineLGj - 
MaineLG.) + Y04 ( VermontSMj - VermontSM.) + Jos ( VermontMEDj - 


VermontMED.) + yo6 ( Vermont LGj - VermontLG.) + uq/ (E. 10 b) 

ply = Yio. (E.lOc) 

p2y = Y20. (E.lOd) 

P3y = Y 30 . (E.lOe) 


A statistically significant and positive coefficient yoi for treatment indicates that the effect of 
the online course on high schools coursetaking is robust when not adjusting for students’ 
mathematics achievement at baseline (that is, excluding state mathematics and pretest 
scores) (table E- 12 ). 

Table E-12. Results of Sensitivity Model 10: Excluding Students’ Prior State 
Mathematics Assessment Scores at Level 1 


95% 

Standard confidence 


Variable 

Logit coefficient 

error 

/;- value 

Odds ratio 

interval 

Intercept 

-0.99 

0.28 

0.001 

0.37 

(0.21,0.65) 

School covariate 

Condition 

0.96 

0.38 

0.014 

2.60 

(1.22,5.53) 

Medium-size school in Maine 

0.30 

0.49 

0.544 

1.35 

(0.51,3.56) 

Large-size school in Maine 

0.38 

0.75 

0.611 

1.47 

(0.33, 6.58) 

Small-size school in Vermont 

0.56 

0.66 

0.398 

1.75 

(0.47, 6.53) 

Medium-size school in Vennont 

-0.47 

0.83 

0.573 

0.62 

(0.12,3.29) 

Large-size school in Vennont 

0.28 

0.69 

0.688 

1.32 

(0.33, 5.30) 

Student covariate 

Female 

-0.12 

0.23 

0.612 

0.89 

(0.57, 1.40) 

Receives special education services 

-1.94 

1.09 

0.077 

0.14 

(0.02, 1.24) 

Eligible for free or reduced-price lunch 

-0.12 

0.27 

0.690 

0.90 

(0.52, 1.54) 


AR is algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Model E.ll 

To examine the relationship between the effect of the intervention on high school coursetaking 
and the covariates specified in the benchmark impact model, model E.ll tests the treatment 
effect while excluding students’ state mathematics assessment scores and their demographic 
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characteristics (gender, eligibility for free or reduced priced lunch, special education status) at 
Level 1 and state by size blocking variables (used for random assignment) at Level 2: 

% = Pq/ (E.lla) 

where % = log (cp,y / 1 - cp ( y) 

Poy = Yoo + Yoi TRTj + Uq/ (E. 1 1 b) 

A statistically significant and positive coefficient y 0 i for treatment indicates that the effect of 
the online course on high school coursetaking is robust when not adjusting for student-level 
variability in baseline mathematics achievement and demographic characteristics and 
school-level variability across size and state (table E-13). 


Table E-13. Results of Sensitivity Model 11: Excluding Covariates at Level 1 or Level 2 


Variable 

Logit coefficient 

Standard 

error 

/?-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

-0.90 

0.26 

0.001 

0.41 

(0.24, 0.68) 

School covariate 

Condition 

0.89 

0.35 

0.014 

2.44 

(1.21,4.92) 


AR is algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced. 

Source: Coursetaking data collected from high schools study students attended in 2009/10. 


Model E.12 

To examine whether the effect of the intervention on high school coursetaking is sensitive to the 
missing data approach, model E.12 tests the benchmark impact model using observed 
(nonimputed) data only, excluding cases that were missing data for high school coursetaking or 
one or more covariates (3.6% of all cases). The loss of data reduces the size of the available 
sample and associated statistical power and may introduce bias into the parameter estimates. To 
reduce potential bias, researchers included the same covariates in the benchmark impact model 
(which also predicted missingness). As was the case in the benchmark analysis, all covariates 
except the treatment indicator were centered on the grand mean. 


T| ij = pq/+ Piy (StateMathj - StateMath..) + P 2 j (SpEd,, - SpEd..) + pay ( FRLj - FRL..) + Pq / 
(. Femaleij - Female..) (E. 1 2a) 

where % = log (cp, 7 / 1 - (p, 7 ) 

poy = Yoo + Yoi TRTj + yo 2 (. MaineMEDj - MaineMED .) + yo 3 ( MaineLGj - 

MaineLG .) + Y 04 ( VermontSMj - VermontSM.) + yos ( VermontMEDj - 

VermontMED .) + Yo6 ( VermontLGj - VermontLG .) + Uq/ (E. 12b) 

P 1/ = Yio (E.12c) 

p2y = Y20 (E.12d) 
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$3/ - Y30 
(fi/ = Y40. 


(E.12e) 

(E.12f) 


A statistically significant and positive coefficient Yoi for treatment indicates that the effect of 
the online course on high school coursetaking is robust to the method of handling missing 
data (multiply imputed versus observed only) (table E-14). 

Table E-14. Results of Sensitivity Model 12: Including Observed (Nonimputed) Data 
Only 

95% 

Standard confidence 


Variable 

Logit coefficient 

error 

/?-value 

Odds ratio 

interval 

Intercept 

-1.20 

0.30 

<0.001 

0.30 

(0.16, 0.55) 

School covariate 

Condition 

1.17 

0.40 

0.005 

3.22 

(1.46, 7.12) 

Medium-size school in Maine 

0.35 

0.51 

0.492 

1.42 

(0.52, 3.90) 

Large-size school in Maine 

0.60 

0.78 

0.443 

1.82 

(0.39, 8.55) 

Small-size school in Vermont 

0.77 

0.68 

0.264 

2.16 

(0.55, 8.44) 

Medium-size school in Vennont 

-0.35 

0.86 

0.686 

0.70 

(0.13,3.93) 

Large-size school in Vennont 

0.10 

0.72 

0.889 

1.11 

(0.26, 4.70) 

Student covariate 

Female 

-0.20 

0.25 

0.416 

0.82 

(0.51, 1.33) 

Receives special education services 

-2.13 

1.14 

0.062 

0.12 

(0.01, 1.12) 

Eligible for free or reduced-price lunch 

0.09 

0.28 

0.749 

1.09 

(0.63, 1.90) 

State mathematics score (standardized) 3 

1.26 

0.24 

<0.001 

3.51 

(2.21, 5.58) 


AR is algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 424 AR students (210 treatment, 214 control); 16 students 
were missing data on the outcome or 1 covariate. The sample size of 424 is different from the analytic sample of 440 because 
this table represents a sensitivity analysis using only observed (nonimputed) data. Coursetaking sequences were coded as 
advanced or not advanced. 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Coursetaking data collected from high schools participating students attended in 2009/10. 


Model E.13 

High school coursetaking outcomes for treatment students in schools that share an online teacher 
may be affected in similar ways by specific qualities and background of the online teacher (for 
example, the teacher’s knowledge or experience teaching online) and are not independent. To 
examine whether the observed treatment effect on high school coursetaking in the benchmark 
analysis is robust to the clustering of students within online teachers, researchers tested a model 
similar to the model estimated for the benchmark analysis, with two exceptions. First, treatment 
students were clustered at Level 2 within the eight online Algebra I teachers, and control students 
were clustered within their standard schools. Second, because school-level covariates (school 
state and size) do not apply to online teachers, only the treatment indicator was included in this 
model at Level 2. As was the case in the benchmark analysis, all covariates except the treatment 
indicator were centered on the grand mean. 
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r\ij = pq/+ Pi/ (StateMathj - StateMath..) + P^ ( SpEdjj - SpEd..) + Pa, ( FRLy - FRL..) + p 4y 


(Femaleij - Female..) (E.13a) 

where r\ , y = log (cp, 7 / 1 - cp (/ ) 

Po/ = Too + Yoi TRTj + u 0/ (E. 1 3b) 

Py = Yio (E.13c) 

p2, = Y2o (E.13d) 

p 3i = Y 3 o (E.13e) 

p4, = Y40. (E.13f) 


A statistically significant and positive coefficient Yoi for treatment indicates that the effect of 
the online course on high school coursetaking is robust to the unit of clustering at Level 2 for 
treatment students by using online teacher instead of school (table E- 15). 


Table E-15. Results of Sensitivity Model 13: Pseudo-School Level 2 Clusters in 
Estimation of AR Students’ High School Coursetaking 


Variable 

Logit coefficient 

Standard 

error 

/;- value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

-1.05 

0.26 

<0.001 

0.35 

(0.21,0.59) 

School covariate 

Condition 

1.08 

0.45 

0.020 

2.94 

(1.20, 7.22) 

Student covariate 

Female 

-0.14 

0.23 

0.555 

0.87 

(0.56, 1.37) 

Receives special education services 

-1.85 

1.08 

0.088 

0.16 

(0.02, 1.32) 

Eligible for free or reduced-price lunch 

0.05 

0.28 

0.867 

1.05 

(0.60, 1.82) 

State mathematics score (standardized) 3 

1.07 

0.21 

<0.001 

2.91 

(1.91,4.44) 


AR is algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 440 AR students (218 treatment, 222 control). Estimates were 
averaged across 10 multiply imputed datasets. Coursetaking sequences were coded as advanced or not advanced, 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

Source: Coursetaking data collected from high schools participating students attended in 2009/10. 
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Impact of Intervention on N-AR Students' Test Scores 

To determine the robustness of the impact estimates based on the benchmark impact model of 
N-AR students algebra scores, researchers conducted the following sensitivity analyses: 

• Model E. 14 includes prior state mathematics assessment scores and study- 
administered pretest scores at Level 1 (grand mean centered). 

• Model E. 15 replaces students’ prior state mathematics assessment scores with their 
study-administered pretest scores at Level 1 (grand mean centered). 

• Model E. 16 does not adjust for baseline mathematics achievement by excluding 
students’ prior state mathematics assessment scores at Level 1. 

• Model E. 17 A excludes all covariates at Level 1 and Level 2. 

• Model E. 18 tests the benchmark impact model on observed (nonimputed) data only. 

All of the sensitivity analyses are consistent with the results of the benchmark impact analysis 
model, where the estimated impact was 0.96 (p = 0.443, effect size = 0.06) (table E-16). In no 
analysis was the effect on N-AR students’ algebra scores at the end of grade 8 statistically 
significant. 


Table E-16. Summary of Sensitivity Analyses Testing Robustness of Effect of 
Condition on N-AR Students’ Algebra Posttest Scores 


Model 

Estimated impact 
(standard error) 

/j-value 

Effect size 

E.14 

includes state mathematics and study-administered pretest 
scores as covariates at Level 1 

0.58 

(1.18) 

0.622 

0.04 

E.15 

replaces state mathematics with study-administered pretest 
scores as covariates at Level 1 

0.39 

(1.12) 

0.726 

0.03 

E.16 

excludes state mathematics and study-administered pretest 
scores as covariates at Level 1 

0.97 

(1.24) 

0.435 

0.06 

E.17 

excludes all covariates at Level 1 and Level 2 

0.61 

(1.36) 

0.655 

0.04 

E.18 

tests benchmark impact model using observed 
(nonimputed) data only 

1.00 

(1.13) 

0.379 

0.07 a 

AR is algebra ready. N-AR is not algebra ready. 





Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates for all models except E.18 were averaged across 10 multiply imputed datasets. 

Model E.18 included 68 schools (35 treatment, 33 control) and 1,199 N-AR students (611 treatment, 588 control); 246 students 
were missing data on the outcome or 1 covariate (estimates are based on observed data only). The sample size of 1,199 is 
different from the analytic sample of 1,445 because this model tests a sensitivity analysis using only observed (not imputed) 
data. Effect sizes were calculated using a pooled standard deviation of the outcome for N-AR students in treatment and control 
schools that incorporates both within and between imputation variance (16.94). 

a. The Eledges’ g effect size was calculated using a pooled standard deviation of the outcome for N-AR students in treatment 
and control schools (13.77).. 

Source: Algebra scores on study-administered Promise Assessment posttests. 
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Model E.14 


Model E. 14 is identical to the model estimated for the benchmark analysis, except that students’ 
study-administered pretest scores were added as an additional adjustment for baseline 
mathematics achievement at Level 1 . As was the case in the benchmark analysis, all covariates 
except the treatment indicator were centered on the grand mean. 


Algebra^- = Pq/ + Pi/ ( StateMathj - StateMath..) + P 2 j ( Pretest] - Pretest..) + P 3 j (SpEd,, - 
SpEd..) + p 4 j(FRLij- FRL..) + \)^(FemaIe, l - Female..) + s,y, (E.14a) 


Pq/ = Yoo + Yoi TRTj + Y 02 ( MaineMEDj - MaineMED.) + 703 ( MaineLGj - 


MaineLG.) + Y 04 ( VermontSMj - VermontSM.) + y 0 5 ( VermontMEDj - 


VermontMED.) + y 0 6 ( Vermont LGj - 

VermontLG.) + uq/. (E. 14b) 

Ply = Yio. 

(E.14c) 

-TO 

II 

tY 

o 

(E.14d) 

-no 

II 

o 

(E.14e) 

o 

II 

GO. 

(E.14f) 

P4y = Y40. 

(E.14g) 

-CD 

II 

o 

(E.14h) 

A nonsignificant coefficient y 0 i for condition indicates that the effect of the online course 
algebra scores is not statistically significant, as is the case in the benchmark model (table 


17). 
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Table E-17. Results of Sensitivity Model 14: Adding Study-Administered Pretest Scores 
at Level 1 (Grand Mean Centered) 


Variable 

Coefficient 

Standard error 

/?-value 

Effect size 

Intercept 

429.91 

0.97 

<0.001 


School covariate 

Condition 

0.58 

1.18 

0.622 

0.04 

Medium-size school in Maine 

-4.77 

1.86 

0.013 


Large-size school in Maine 

-5.53 

2.32 

0.020 


Small-size school in Vermont 

-8.20 

2.88 

0.045 


Medium-size school in Vermont 

-5.87 

2.88 

0.045 


Large-size school in Vermont 

-5.14 

2.21 

0.024 


Student covariates 

Female 

-1.92 

0.89 

0.032 


Receives special education services 
Eligible for free or reduced-price 

-2.04 

1.49 

0.177 


lunch 

-0.94 

0.85 

0.270 


Pretest 

0.09 

0.02 

<0.001 


State mathematics score 
(standardized) 3 

2.57 

0.71 

0.001 




Chi-squared (degrees of 




Variance 

freedom) 

p-value 


Residual 

210.53 




Level 2 (school) 

6.40 

97.28 (57) 

0.001 


Total variance 

216.93 





N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was calculated 
using a pooled standard deviation of the outcome for N-AR students in treatment and control schools that incorporates both 
within and between imputation variance (16.94). 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 

Model E.15 

Model E.15 is identical to the model estimated for the benchmark analysis, except that students’ 
state mathematics assessment scores were replaced with their scores on the study-administered 
pretest at Level 1 . As was the case in the benchmark analysis, all covariates except the treatment 
indicator were centered on the grand mean. 


Algebra jj = Pq/ + Pi/ {Pretest y - Pretest.) + (Lj ( SpEdy - SpEd.) + [E,, ( ERLy - FRL.) + p 4y - 
(. Femalejj - Female.) + s ,y. (E. 15a) 

Pq/ = Too + Yoi TRTj + 702 ( MaineMEDj - MaineMED) + Y 03 ( MaineLGj - 
MaineLG) + yo 4 ( VermontSMj - VermontSM) + yos ( VermontMEDj - 


VermontMED) + yo 6 ( VermontLGj - VermontLG) + uq/ (E. 15b) 

Py = Yio. (E.15c) 

p2y = Y20. (E.15d) 

p3y = Y 30 . (E.15e) 

p4/ = Y40. (E.15f) 
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A nonsignificant coefficient yoi for treatment indicates that the effect of the online course on 
algebra scores is not statistically significant, as is the case in the benchmark model (table E- 
18 ). 

Table E-18. Results of Sensitivity Model 15: Replacing Students’ Prior State 
Mathematics Assessment Scores with Study-Administered Pretest Scores at Level 1 (Grand 
Mean Centered) 


Variable 

Coefficient 

Standard error 

/;- value 

Effect size 

Intercept 

430.10 

0.95 

<0.001 


School covariate 

Condition 

0.39 

1.12 

0.726 

0.03 

Medium-size school in Maine 

-5.13 

1.84 

0.008 


Large-size school in Maine 

-6.20 

2.23 

0.008 


Small-size school in Vermont 

-8.95 

2.86 

0.003 


Medium-size school in Vermont 

-6.67 

2.77 

0.019 


Large-size school in Vermont 

-5.60 

2.16 

0.012 


Student covariate 

Female 

-1.85 

0.89 

0.039 


Receives special education services 
Eligible for free or reduced-price 

-3.18 

1.45 

0.033 


lunch 

-1.25 

0.85 

0.142 


Pretest 

0.13 

0.02 

<0.001 



Variance 

Chi-squared (degrees of 
freedom) 

/?- value 


Residual 

214.52 




Level 2 (school) 

4.86 

88.14(57) 

0.005 


Total variance 

219.38 





N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was calculated 
using a pooled standard deviation of the outcome for N-AR students in treatment and control schools that incorporates both 
within and between imputation variance (16.94). 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Model E.16 

Model E. 16 is identical to the model estimated for the benchmark analysis, except that 
researchers excluded students’ state mathematics assessment scores at Level 1 and controlled 
only for student demographic characteristics (gender, eligibility for free or reduced priced lunch, 
special education status) at Level 1 and state by size blocking variables (used for random 
assignment) at Level 2 . As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


Algebra^- = |3q/ + (3ij ( SpEdj - SpEd..) + P 2 j ( FRLjj - FRL..) + [L,, ( Female y - Female..) + 
s ij. (E. 16 a) 

P07 = Too + Y01 TRTj + 702 ( MaineMEDj - MaineMED .) + 703 ( MaineLGj - 

MaineLG .) + 704 ( VermontSMj - VermontSM.) + 705 ( VermontMEDj - 
VermontMED.) + 706 ( VermontLGj - VermontLG .) + Uq/. (E. 16 b) 
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(E.16c) 

(E.16d) 

(E.16e) 


Piy = Yio. 

p2/ = Y20. 

P 3 / = Y30. 

A nonsignificant coefficient Y 01 for treatment indicates that the effect of the online course on 
algebra scores is not statistically significant, as is the case in the benchmark model (table E- 
19). 

Table E-19. Results of Sensitivity Model 16: Excluding Students’ Prior State 
Mathematics Assessment Scores at Level 1 


Variable 

Coefficient 

Standard error 

/?-value 

Effect size 

Intercept 

430.03 

1.02 

<0.001 


School covariate 

Condition 

0.97 

1.24 

0.435 

0.06 

Medium-size school in Maine 

-5.60 

1.92 

0.005 


Large-size school in Maine 

-7.33 

2.37 

0.003 


Small-size school in Vermont 

-9.21 

2.97 

0.003 


Medium-size school in Vermont 

-7.18 

2.93 

0.018 


Large-size school in Vermont 

-5.91 

2.31 

0.014 


Student covariate 

Female 

-1.56 

0.91 

0.088 


Receives special education services 
Eligible for free or reduced-price 

-6.04 

1.37 

<0.001 


lunch 

-2.25 

0.85 

0.009 



Variance 

Chi-squared (degrees 
of freedom) 

/7-value 


Residual 

224.10 




Level 2 (school) 

7.23 

97.06 (57) 

0.001 


Total variance 

231.33 




N-AR is not algebra ready. 


Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was calculated 
using a pooled standard deviation of the outcome for N-AR students in treatment and control schools at posttest that incorporates 
both within and between imputation variance (16.94). 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Model E.17 

To examine the relationship between the effect of the intervention on algebra scores and the 
covariates specified in the benchmark impact model, model E.17 tests the treatment effect while 
excluding students’ state mathematics assessment scores and their demographic characteristics 
(gender, eligibility for free or reduced priced lunch, special education status) at Level 1 and state 
by size blocking variables (used for random assignment) at Level 2. 

Algebra^ = p 0/ + Sy. (E. 1 7a) 

Pq/ = Yoo + Yoi TRTj + uq/. (E. 1 7b) 
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A nonsignificant coefficient yoi for treatment indicates that the effect of the online course on 
algebra scores is not statistically significant, as is the case in the benchmark model (table E- 
20 ). 

Table E-20. Results of Sensitivity Model 17: Excluding Covariates at Level 1 or Level 2 


Variable 

Coefficient 

Standard error 

/7-value 

Effect size 

Intercept 

429.20 

1.00 

<0.001 


School covariate 

Condition 

0.61 

1.36 

0.655 

0.04 



Chi-squared (degrees 




Variance 

of freedom) 

/7-value 


Residual 

232.31 




Level 2 (school) 

10.77 

127.45 (62) 

<0.001 


Total variance 

243.08 





N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. The effect size was calculated 
using a pooled standard deviation of the outcome for N-AR students in treatment and control schools at posttest that incorporates 
both within and between imputation variance (16.94). 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Model E.18 

To examine whether the effect of the intervention on algebra posttest scores is sensitive to the 
missing data approach, model E.18 tests the benchmark impact model using observed 
(nonimputed) data only, excluding cases that were missing data for the algebra posttest or any 
covariate in the model (17% of all cases). The loss of data reduces the size of the available sample 
and associated statistical power and may introduce bias into the parameter estimates. To reduce 
potential bias, researchers included the same covariates in the benchmark impact model (which 
also predicted missingness). As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


Algebra^ = |3q/ + Pi j(StateMathj - StateMath..) + [fi, ( SpEdj - SpEd..) + p 4/ (FRL, f - 
FRL..) + |3 4/ (Female^ - Female . .) + e,y. (E. 1 8a) 

poy = Too + Yoi TRTj + 702 ( MaineMEDj - MaineMED .) + yo 3 ( MaineLGj - 

MaineLG .) + yp 4 ( VermontSMj - VermontSM.) + Jos ( VermontMEDj - 


VermontMED.) + yo6 ( VermontLGj - VermontLG.) + Uq/. (E. 18b) 

Pi./ = Yio. (E.18c) 

P 2/ = Y 20 . (E.18d) 

p 3 / = Y 30 . (E.18e) 

p4/ = Y4o. (E.18f) 
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A nonsignificant coefficient yoi for treatment indicates that the effect of the online course on 
algebra scores is not statistically significant, as is the case in the benchmark model (table E- 
21). 

Table E-21. Results of Sensitivity Model 18: Including Observed (Nonimputed) Data 
Only 


Variable 

Coefficient 

Standard error 

/?- value 

Effect size 

Intercept 

429.66 

0.92 

<0.001 


School covariate 

Condition 

1.00 

1.13 

0.379 

0.06 a 

Medium-size school in Maine 

-4.47 

1.78 

0.015 


Large-size school in Maine 

-4.97 

2.12 

0.023 


Small-size school in Vermont 

-7.53 

2.80 

0.010 


Medium-size school in Vermont 

-6.66 

2.77 

0.020 


Large-size school in Vermont 

— 4.47 

2.06 

0.034 


Student covariate 

Female 

-1.38 

0.86 

0.109 


Receives special education services 
Eligible for free or reduced-price 

-2.06 

1.28 

0.106 


lunch 

State mathematics score 

-1.12 

0.87 

0.198 


(standardized) 13 

4.44 

0.56 

<0.001 



Variance 

Chi-squared (degrees 
of freedom) 

/?- value 


Residual 

211.61 




Level 2 (school) 

4.65 

85.02 (56) 

0.008 


Total variance 

216.26 





N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 1,199 N-AR students (61 1 treatment, 588 control); 4 control 
schools had no N-AR students; 246 students were missing data on the outcome or 1 covariate. The sample size of 1,199 is 
different from the analytic sample of 1,445 because this table represents a sensitivity analysis using only observed (nonimputed) 
data. 

a. The Hedges’ g effect size was calculated using a pooled standard deviation of the outcome for N-AR students in treatment 
and control schools (13.77). 

b. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

Source: Algebra scores on study-administered Promise Assessment posttest. 


Impact of Intervention on N-AR Students’ Planned Grade 9 Coursetaking 

To determine the robustness of the impact estimates based on the benchmark impact model of 
N-AR students’ planned grade 9 coursetaking, researchers conducted the following sensitivity 
analyses: 

• Model E. 19 includes prior state mathematics assessment scores and study- 
administered pretest scores at Level 1 (grand mean centered). 

• Model E.20 replaces students’ prior state mathematics assessment scores with their 
study-administered pretest scores at Level 1 (grand mean centered). 

• Model E.2 1 does not adjust for baseline mathematics achievement by excluding 
students’ prior state mathematics assessment scores at Level 1. 


E25 


• Model E.22 A excludes all covariates at Level 1 and Level 2. 

• Model E.23 tests the benchmark impact model on observed (nonimputed) data only. 

The results of the sensitivity analyses presented in table E-22 are consistent with the results of 
the benchmark impact analysis model, where the estimated logit coefficient was 0.78 (p = 0.099, 
odds ratio = 2.18), with one exception. In model E.23, which tested the benchmark impact model 
using observed data only (dropping 10% of cases for which N-AR students were missing 
planned coursetaking or one or more covariates at Level 1), the treatment effect became 
significant (p = 0.035, using an alpha value of 0.05), indicating that N-AR students from 
treatment schools were more likely to enroll in a grade 9 mathematics course at or above Algebra 
I than N-AR students in control schools. In each of the other sensitivity analyses, the effect of 
the intervention on N-AR students’ planned school coursetaking was not statistically significant. 


Table E-22. Summary of Sensitivity Analyses Testing Robustness of Effect of 
Condition on N-AR Students’ Planned Grade 9 Coursetaking 


Model 

Logit coefficient 
(standard error) 

/7-value 

Odds ratio 
(95% 
confidence 
interval) 

E.19 

0.71 

0.135 

2.03 

includes state mathematics and study-administered 
pretest scores as covariates at Level 1 

(0.47) 


(0.80, 5.17) 

E.20 

0.62 

0.138 

1.87 

replaces state mathematics with study-administered 
pretest scores as covariates at Level 1 

(0.41) 


(0.81,4.28) 

E.21 

0.69 

0.074 

2.00 

excludes state mathematics and study-administered 
pretest scores as covariates at Level 1 

(0.38) 


(0.93,4.28) 

E.22 

0.34 

0.361 

1.40 

excludes all covariates at Level 1 and Level 2 

(0.37) 


(0.68,2.91) 

E.23 

1.06 

0.035 

2.90 

tests benchmark impact model using observed 
(nonimputed) data only 

(0.49) 


(1.80, 7.75) 


N-AR is not algebra ready. 

Note : Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates for all models except E.23 were averaged across 10 multiply imputed datasets. Model 
E.23 included 68 schools (35 treatment, 33 control) and 1,298 N-AR students (672 treatment, 626 control); 147 students were 
missing data on the outcome or 1 covariate (estimates are based on observed data only). The sample size of 1,298 is different 
from the analytic sample of 1,445 because this model tests a sensitivity analysis using only observed (nonimputed) data. Planned 
courses were coded as representing a course at or above Algebra I or not at or above Algebra I. 

Source: Planned courses indicated by study students at end of Grade 8. 
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Model E.19 


Model E. 19 is identical to the model estimated for the benchmark analysis, except that students’ 
study-administered pretest scores is added as an additional adjustment for baseline mathematics 
achievement at Level 1 . As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


T | ij = po/+ Pi / ( StateMathj - StateMath..) + Pu f Pretest j - Pretest..) + P 3 j (SpEd,, - SpEd..) 
+ P 4 / ( FRLjj - FRL . .) + Ps, ( Female# - Female.) (E. 1 9a) 


where % = log (cp,y / 1 - (p ;/ ) 


Pq/ = Yoo + Y 01 TRTj + Y 02 ( MaineMEDj - MaineMED.) + Y 03 ( MaineLGj - 


MaineLG .) + Y 04 ( VermontSMj - VermontSM) + Y 05 ( Vermont M ED j - 


VermontMED.) + yo6 ( VermontLG / - VermontLG.) + uqy 

(E.19b) 

Ply = Y 10 

(E.19c) 

P2y = Y20 

(E.19d) 

P3y = Y30 

(E.19e) 

P 47 = Y40 

(E.19f) 

P5y = Y50. 

(E.19g) 


A nonsignificant coefficient Y 01 for condition indicates that the effect of the online course on 
planned grade 9 coursetaking is not statistically significant, as is the case in the benchmark 
model (table E-23). 
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Table E-23. Results of Sensitivity Model 19: Adding Study-Administered Pretest Scores 
at Level 1 (Grand Mean Centered) 


Variable 

Logit coefficient 

Standard 

error 

/>-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

1.10 

0.35 

0.003 

3.00 

(1.49,6.05) 

School covariate 

Condition 

0.71 

0.47 

0.135 

2.03 

(0.80,5.17) 

Medium-size school in Maine 

1.43 

0.57 

0.015 

4.18 

(1.34, 13.08) 

Large-size school in Maine 

0.82 

0.88 

0.359 

2.26 

(0.39, 13.25) 

Small-size school in Vermont 

0.12 

0.61 

0.846 

0.56 

(0.12,2.70) 

Medium-size school in Vermont 

-1.74 

0.93 

0.067 

0.18 

(0.03, 1.14) 

Large-size school in Vermont 

0.20 

0.84 

0.817 

1.22 

(0.23, 6.53) 

Student covariate 

Female 

-0.09 

0.18 

0.608 

0.91 

(0.64, 1.30) 

Receives special education services 

-1.24 

0.24 

<0.001 

0.29 

(0.18, 0.46) 

Eligible for free or reduced-price 

-0.53 

0.17 

0.002 

0.59 

(0.43, 0.82) 

Pretest 

0.02 

0.00 

<0.001 

1.02 

(1.01, 1.03) 

State mathematics score 

1.24 

0.19 

<0.001 

3.45 

(2.36, 5.05) 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. Planned courses were coded as 
representing a course at or above Algebra I or not at or above Algebra I. 

a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating in 
the study. 

Source: Planned courses indicated by study students at end of Grade 8. 


Model E.20 

Model E.20 is identical to the model estimated for the benchmark analysis, except that students’ 
state mathematics assessment scores were replaced with their scores on the study-administered 
pretest at Level 1 . As was the case in the benchmark analysis, all covariates except the treatment 
indicator were centered on the grand mean. 


r\y = ( 3 o/+ Pi, C Pretesty - Pretest.) + ( 3 2 j ( SpEdy - SpEd.) + $y(FRLjj - FRL.) + ( 3 4/ 

( Femaley - Female.) (E.20a) 


where % = log (cp ;> / 1 - cp (/ ) 


J 3 o/ = Yoo + Yoi TRTj + 702 ( MaineMEDj - MaineMED ) + 703 ( MaineLGj - 
MaineLG) + yo 4 ( VermontSMj - VermontSM) + Jos ( VermontMEDj - 


VermontMED) + yo6 ( VermontLGj - VermontLG) + Uq/ (E.20b) 

Pi/ = Yio (E.20c) 

p 2 , = Y 20 (E.20d) 

P 37 = Y 30 (E.20e) 

p 4 / = y 4 o. (E.20f) 
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A nonsignificant coefficient Yoi for treatment indicates that the effect of the online course on 
planned grade 9 coursetaking not statistically significant, as is the case in the benchmark 
model (table E-24). 

Table E-24. Results of Sensitivity Model 20: Replacing Students’ Prior State 
Mathematics Assessment Scores with their Study-Administered Pretest Scores at Level 1 
(Grand Mean Centered) 


Variable 

Logit coefficient 

Standard 

error 

/?-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

1.00 

0.30 

0.002 

2.72 

(1.50,4.95) 

School covariate 

Condition 

0.62 

0.41 

0.138 

1.87 

(0.81,4.28) 

Medium-size school in Maine 

1.11 

0.52 

0.038 

3.02 

(1.07,8.57) 

Large-size school in Maine 

0.44 

0.79 

0.579 

1.56 

(0.32,7.56) 

Small-size school in Vermont 

-0.94 

0.70 

0.188 

0.39 

(0.10, 1.59) 

Medium-size school in Vennont 

-1.96 

0.84 

0.023 

0.14 

(0.03, 0.75) 

Large-size school in Vermont 

-0.07 

0.73 

0.926 

0.93 

(0.22,4.03) 

Student covariate 

Female 

-0.02 

0.16 

0.877 

0.98 

(0.71, 1.34) 

Receives special education services 

-1.63 

0.21 

<0.001 

0.20 

(0.13,0.30) 

Eligible for free or reduced-price 






lunch 

-0.61 

0.16 

<0.001 

0.54 

(0.40, 0.74) 

Pretest 

0.03 

0.00 

<0.001 

1.03 

(1.03, 1.04) 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control), and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. Planned courses were coded as 
representing a course at or above Algebra I or not at or above Algebra I. 

Source: Planned courses indicated by study students at end of Grade 8. 


Model E.21 

Model E.2 1 is identical to the model estimated for the benchmark analysis, except that 
researchers excluded students’ state mathematics assessment scores at Level 1 and controlled 
only for student demographic characteristics (gender, eligibility for free or reduced priced lunch, 
special education status) at Level 1 and state by size blocking variables (used for random 
assignment) at Level 2. As was the case in the benchmark analysis, all covariates except the 
treatment indicator were centered on the grand mean. 


T | y = po/ + (3 1 j ( SpEdj - SpEd. .) + [A/ ( FRLy - FRL . .) + (A, ( Femaley - Female..) 

(E.2 la) 

where % = log (cp, 7 / 1 - cp (/ ) 

(Ay = Yoo + Yoi TRTj + Y02 ( MaineMEDj - MaineMED .) + yo3 ( MaineLGj - 
MaineLG.) + Y 04 ( VermontSM ) - VermontSM .) + Y 05 ( VermontMEDj - 


VermontMED.) + yo6 ( VermontLGj - VermontLG .) + uq/ (E.21b) 

(A/ = Y 10 (E.21c) 

(A, = Y 20 (E.2 Id) 

(3 3/ = Y 3 (). (E.21e) 
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A nonsignificant coefficient yoi for treatment indicates that the effect of the online course on 
planned grade 9 coursetaking is not statistically significant, as is the case in the benchmark 
model (table E-25). 

Table E-25. Results of Sensitivity Model 21: Excluding Students’ Prior State 
Mathematics Assessment Scores at Level 1 


Variable 

Logit coefficient 

Standard 

error 

/;-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

0.85 

0.27 

0.003 

2.33 

(1.35,4.03) 

School covariate 

Condition 

0.69 

0.38 

0.074 

2.00 

(0.93,4.28) 

Medium-size school in Maine 

0.89 

0.48 

0.068 

2.44 

(0.93, 6.36) 

Large-size school in Maine 

0.15 

0.72 

0.832 

1.17 

(0.27, 4.97) 

Small-size school in Vermont 

-0.87 

0.66 

0.189 

0.42 

(0.11, 1.55) 

Medium-size school in Vennont 

-1.79 

0.77 

0.023 

0.17 

(0.04, 0.77) 

Large-size school in Vermont 

-0.10 

0.67 

0.883 

0.91 

(0.24, 3.46) 

Student covariate 

Female 

0.00 

0.15 

0.992 

1.00 

(0.75, 1.33) 

Receives special education services 

-2.07 

0.20 

<0.001 

0.13 

(0.09,0.19) 

Eligible for free or reduced-price 






lunch 

-0.80 

0.15 

<0.001 

0.45 

(0.33, 0.60) 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. Planned courses were coded as 
at or above Algebra I or not at or above Algebra I. 

Source: Planned courses indicated by study students at end of Grade 8. 


Model E.22 

To examine the relationship between the effect of the intervention on planned grade 9 
coursetaking and the covariates specified in the benchmark impact model, model E.22 tested the 
treatment effect while excluding students’ state mathematics assessment scores and their 
demographic characteristics (gender, eligibility for free or reduced priced lunch, special 
education status) at Level 1 and state by size blocking variables (used for random assignment) at 
Level 2. 


fiy Po, 

where % = log (cp,y / 1 - cpy) 

(E.22a) 

Pq/ = yoo + yoi TRTj + uo/ 

(E.22b) 


A nonsignificant coefficient y 0 i for treatment indicates that the effect of the online course on 
planned grade 9 coursetaking is the same as in the benchmark model (table E-26). 
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Table E-26. Results of Sensitivity Model 22: Excluding Covariates at Level 1 or Level 2 


Variable 

Logit coefficient 

Standard 

error 

/7-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

0.86 

0.26 

0.002 

2.35 

(1.39,3.99) 

School covariate 






Condition 

0.34 

0.37 

0.361 

1.40 

(0.68,2.91) 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,445 N-AR students (744 treatment, 701 control); 4 control 
schools had no N-AR students. Estimates were averaged across 10 multiply imputed datasets. Planned courses were coded as at 
or above Algebra I or not at or above Algebra I. 

Source: Planned courses indicated by study students at end of Grade 8. 


Model E.23 

To examine whether the effect of the intervention on planned grade 9 coursetaking is sensitive 
to the missing data approach (multiple imputation, details provided in appendix F), model E.23 
tests the benchmark impact model using the observed (nonimputed) data only, excluding cases 
that were missing data for planned grade 9 coursetaking or any covariate in the model (10%). 
The loss of data reduces the size of the available sample and associated statistical power and may 
introduce bias into the parameter estimates. To reduce potential bias, researchers included the 
same covariates in the benchmark impact model (which also predicted missingness). As was the 
case in the benchmark analysis, all covariates except the treatment indicator were centered on the 
grand mean. 


T | ij = pq/+ Pi/ (StateMath j - StateMath..) + Pu (SpEd,, - SpEd..) + Pay ( FRL/j - FRL..) + P^ 
( Femaleij - Female.) (E.23a) 

where % = log (cp /y / 1 - cp (/ ) 

poy = Yoo + Yoi TRTj + 702 ( MaineMEDj - MaineMED .) + 703 (MaineLGj - 
MaineLG .) + yo 4 ( VermontSMj - VermontSM.) + Jos ( VermontMEDj - 


VermontMED) + yo6 ( VermontLGj - VermontLG.) + Uq/ (E.23b) 

Pi/ = Yio (E.23c) 

P 2 / = Y 20 (E.23d) 

p 3 / = Y 3 o (E.23e) 

p4/ = Y4o. (E.23f) 


A significant and positive coefficient yoi for treatment indicates that, unlike in the 
benchmark model, N-AR students in treatment schools were more likely to enroll in an 
intermediate course sequence than N-AR students in control schools (table E-27). 
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Table E-27. Results of Sensitivity Model 23: Observed (Nonimputed) Data Only 


Variable 

Logit coefficient 

Standard 

error 

/7-value 

Odds ratio 

95% 

confidence 

interval 

Intercept 

0.99 

0.35 

0.008 

2.68 

(1.32,5.45) 

School covariate 

Condition 

1.06 

0.49 

0.035 

2.90 

(1.08,7.75) 

Medium-size school in Maine 

1.47 

0.64 

0.026 

4.35 

(1.20, 15.70) 

Large-size school in Maine 

0.83 

0.97 

0.397 

2.28 

(0.33, 15.78) 

Small-size school in Vermont 

-1.18 

0.85 

0.168 

0.31 

(0.06, 1.67) 

Medium-size school in Vermont 

-2.35 

1.00 

0.023 

0.10 

(0.01,0.71) 

Large-size school in Vermont 

0.13 

0.89 

0.884 

1.14 

(0.19, 6.78) 

Student covariate 

Female 

-0.17 

0.18 

0.347 

0.85 

(0.60, 1.20) 

Receives special education services 

-1.11 

0.25 

<0.001 

0.33 

(0.20, 0.54) 

Eligible for free or reduced-price 






lunch 

-0.49 

0.18 

0.006 

0.61 

(0.44, 0.87) 

State mathematics score 

2.01 

0.16 

<0.001 

7.43 

(5.48, 10.07) 


N-AR is not algebra ready. 

Note: Sample includes 68 schools (35 treatment, 33 control) and 1,298 N-AR students (672 treatment, 626 control); 4 control 
schools had no N-AR students; 147 students were missing data on the outcome or 1 covariate. The sample size of 1,298 is 
different from the analytic sample of 1,445 because this table represents a sensitivity analysis using only observed (not 
imputed) data. Planned courses were coded as at or above Algebra I or not at or above Algebra I. 

a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

Source: Planned courses indicated by study students at end of Grade 8. 


Although the result from Model 23 varied from the benchmark analysis result, both models 
were conducted to test a secondary research question about whether the offering of online 
Algebra I to AR students had any significant negative side effects on N-AR students. In all 
of the sensitivity analyses (including Model 23), the results indicated that the effect of the 
intervention was not significantly negative on N-AR students’ planned coursetaking. 
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APPENDIX F: MISSING DATA AND MULITPLE IMPUTATION 


This appendix describes the approach used for handling missing data. It describes the rates and 
patterns of missing data in the study, analyses conducted to identify student- and school-level 
predictors of missing data, the models used for multiple imputation, and provides graphical and 
numeric diagnostics that display the results of the imputation. 

Imputation Sample 

Table F-l displays the rates of missing data for both covariates and outcome measures. The rates 
of missing data were not statistically significantly different by treatment status. 


Table F-l. Percentage of Missing Cases on Measures in Impact Analyses 




AR students 



N-AR students 



Total 

Treatment 

Control 

P- 

Total 

Treatment 

Control 

P- 

Variable 

schools 

schools 

value 

schools 

schools 

value 

Covariate 

Pretest 

1 

< 1 

2 

0.298 

4 

4 

4 

0.829 

State mathematics score 

< 1 

< 1 

< 1 

0.582 

3 

3 

3 

0.778 

Gender 

0 

0 

0 


< 1 

< 1 

< 1 

0.466 

Eligible for free or reduced- 
price lunch 

< 1 

< 1 

1 

0.351 

3 

3 

3 

0.982 

Receiving special education 
services 

< 1 

< 1 

< 1 

0.582 

2 

2 

2 

0.808 

Outcome 

Algebra posttest 

1 

< 1 

2 

0.435 

14 

15 

13 

0.737 

General mathematics posttest 

< 1 

< 1 

1 

0.353 

7 

7 

7 

0.846 

Planned grade 9 course 

3 

1 

5 

0.081 

7 

6 

7 

0.996 

High school coursetaking 

3 

3 

3 

0.754 

— 

— 

— 

— 


AR is algebra ready. N-AR is not algebra ready. — is not available. 

Note: Sample includes 68 schools (35 treatment, 33 control); 440 AR students (218 treatment, 222 control); and 1,445 N-AR 
students (744 treatment, 701 control). Estimates represent only missing data that were subsequently imputed. Statistical 
comparisons between treatment and control account for the clustering of students within schools. High school coursetaking data 
were collected only for AR students. 

Source: Study team records. 

Missing Data Patterns 

To identify patterns of missing data, researchers first examined the descriptive statistics for 
students with missing outcome data and those with complete data (tables F-2 through F-5). 


FI 


Table F-2. Descriptive Statistics for AR Students with Missing Algebra Posttest Data and 
AR Students with Complete Data 



Overall 

sample 

Missing 

algebra posttest 
data 

Complete 
algebra posttest 
data 

Characteristic 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Total sample 

Eligible for free or reduced-price lunch 

440 

100 

6 

1 

434 

99 

(percent) 

Receives special education services 

139 

32 

3 

50 

136 

31 

(percent) 

15 

3 

0 

0 

15 

3 

Has limited English proficiency (percent) 

15 

3 

0 

0 

15 

3 

Female (percent) 

214 

49 

<3 

<50 

>211 

>48 

Racial/ethnic minority (percent) 

29 

7 

0 

0 

29 

7 


Number 

Mean 

Number 

Mean 

Number 

Mean 

Grade 7 score on state mathematics 
assessment (standardized) 2 

437 

0.95 

(0.69) 

6 1.16 
(1.07) 

431 

0.95 

(0.68) 

Fall 2008 pretest score (Promise 
Assessment) 15 

435 

349.87 

(23.27) 

5 360.60 

(19.37) 

430 

349.74 

(23.30) 


Note: Sample included 440 students. Observed data were used to populate table. Figures in parentheses are standard deviations. 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

b. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source : Maine state department of education and Vermont supervisory unions, study records. 

Table F-3. Descriptive Statistics for AR Students with Missing High School Coursetaking 
Data and AR Students with Complete Data 


Characteristic 

Overall 

sample 

Missing 
high school 
coursetaking data 

Complete 
high school 
coursetaking data 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Overall 

440 

100 

13 

3 

427 

97 

Eligible for free or reduced-price lunch 







(percent) 

139 

32 

6 

46 

133 

31 

Receives special education services (percent) 

15 

3 

<3 

<23 

> 12 

>2 

Has limited English proficiency (percent) 

15 

3 

<3 

<23 

> 12 

>2 

Female (percent) 

214 

49 

5 

38 

209 

49 

Racial/ethnic minority (percent) 

29 

7 

<3 

<23 

>26 

>6 


Number 

Mean 

Number 

Mean 

Number 

Mean 

Grade 7 score on state mathematics 

437 

0.95 

13 

0.96 

424 

0.96 

assessment (standardized) 2 


(0.69) 


(0.69) 


(0.69) 

Fall 2008 pretest score (Promise 

435 

349.87 

13 

349.94 

422 

347.54 

Assessment) 15 


(23.27) 


(23.50) 


(14.19) 


Note: Sample includes 440 students. Observed data were used to populate table. Figures in parentheses are standard deviations, 
a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 


standardized by using the mean and standard deviation of the test scores within each state, including only schools 
participating in the study. 

b. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source: Maine state department of education and Vermont supervisory unions, study records. 
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Table F-4. Descriptive Statistics for N-AR Students with Missing Algebra Data and N-AR 
Students with Complete Data 



Overall 

sample 

Missing 

algebra posttest 
data 

Complete 
algebra posttest 
data 

Characteristic 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Overall 

1,445 

100 

204 

14 

1,241 

86 

Eligible for free or reduced-price lunch 
(percent) 

670 

46 

127 

62 

543 

44 

Receives special education services(percent) 

252 

17 

58 

28 

194 

16 

Has limited English proficiency (percent) 

43 

3 

11 

5 

32 

3 

Female (percent) 

717 

50 

111 

54 

611 

49 

Racial/ethnic minority (percent) 

75 

5 

18 

9 

57 

5 


Number 

Mean 

Number 

Mean 

Number 

Mean 

Grade 7 score on state mathematics 
assessment (standardized) 2 

1,403 

-0.24 

(0.86) 

191 

-0.55 

(0.95) 

1,212 

-0.19 

(0.84) 

Fall 2008 pretest score (Promise 
Assessment) 15 

1,384 

312.60 

(27.23) 

181 

304.52 

(28.47) 

1,203 

313.81 

(26.84) 


Note: Sample includes 1,445 students. Observed data were used to populate table. Figures in parentheses are standard 
deviations. 

a. Because Maine and Vermont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

b. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source : Maine state department of education and Vermont supervisory unions, study records.. 


Table F-5. Descriptive Statistics for N-AR Students with Missing Planned Grade 9 
Coursetaking Data versus N-AR Students with Complete Data 



Overall 

sample 

Missing 

planned grade 9 
coursetaking data 

Complete 
planned grade 9 
coursetaking data 

Characteristic 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Overall 

Eligible for free or reduced-price lunch 

1,445 

100 

96 

7 

1,349 

93 

(percent) 

670 

46 

69 

72 

601 

45 

Receives special education services(percent) 

252 

17 

29 

30 

223 

17 

Has limited English proficiency (percent) 

43 

3 

8 

8 

35 

3 

Female (percent) 

717 

50 

44 

46 

673 

50 

RaciaEethnic minority (percent) 

75 

5 

9 

9 

66 

5 


Number 

Mean 

Number 

Mean 

Number 

Mean 

Grade 7 score on state mathematics 
assessment (standardized) 2 

1,403 

0.24 

(0.86) 

91 

0.35 

(1.22) 

1,312 

0.23 

(0.83) 

Fall 2008 pretest score (Promise Assessment) 

1,384 

312.60 

(27.23) 

82 

304.85 

(29.00) 

1,302 

313.09 

(27.05) 


Note : Sample includes 1,445 students. Observed data were used to populate table. Figures in parentheses are standard 
deviations. 

a. Because Maine and Vennont use different tests, it was necessary to translate scores into a common metric. The scores were 
standardized by using the mean and standard deviation of the test scores within each state, including only schools participating 
in the study. 

b. The Promise Assessment test was administered in the first month of the school year and is therefore not a pure pretreatment 
measure. 

Source : Maine state department of education and Vennont supervisory unions, study records. 
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Researchers then examined the extent to which student and school baseline characteristics were 
related to missingness. Given limited missing data (l%-3%) for the AR sample and a smaller 
sample size (n = 440), there was not sufficient statistical power to detect missing data patterns. 
Consequently, analyses of missing data patterns are reported only for the N-AR sample (64 
study schools, 1,445 students). 58 

To test whether missing data were predicted by any of the covariates included in the impact 
model and sensitivity analyses, researchers conducted a series of exploratory logistic regression 
analyses. They created four binary outcome variables that captured whether a student was 
missing data on the pretest, the general mathematics posttest, the algebra posttest, or grade 9 
enrollment. For these four variables, a value of 0 represented complete data and a value of 1 
represented missing data. Researchers then regressed the missing data indicators on each 
covariate in the impact and sensitivity models, controlling for treatment status. They ran separate 
models for each covariate (that is, the model for gender included only gender and treatment 
status as predictors of missing pretest) and each missing outcome (that is, separate models were 
tested for missing pretest, missing general mathematics, missing algebra, and missing planned 
grade 9 enrollment data). All significant predictors (and those that approached significance at p < 
.10) were then included in the imputation models. 

Results indicated systematic differences between students missing data and those with complete 
data: 

• Boys were more likely to have missing data than girls. 

• Students eligible for special education services or free or reduced-price lunch were more 
likely to have missing data than those not receiving these services. 

• Students with lower scores on the state mathematics assessment, study-administered 
pretest, general mathematics posttest, and algebra posttest were more likely to have 
missing data (on one or more measures) than students with higher scores. 

• Students at larger schools were more likely to be missing algebra and coursetaking 
posttests than students at medium sized schools. 

• Students at smaller schools were more likely to be missing pretest scores than students at 
medium schools. 

• State (Maine versus Vennont) was not related to missingness. 

Researchers conducted a similar series of logistic regression analyses to identify auxiliary 
variables not specified in the confirmatory impact models that might help predict missingness. 
Student-level predictors of missingness included racial/ethnic minority status, English language 
learner/limited English proficient status, time spent on the pretest, time spent on the general 
mathematics posttest, and time spent on the algebra posttest. 


58 This is the total number of N-AR students in treatment and control schools who were not withdrawn 
from the study by their parents or their schools. (See appendix A, figure A-l, for details about the data 
collection and analysis sample.) 


F4 



To identify school-level auxiliary variables, researchers examined school factors from two 
sources: publicly available data about the schools from the Common Core of Data and sample- 
specific school-level aggregates of N-AR student data. Out of 20 school characteristics 
examined, 16 school-level auxiliary variables predicted missing data and were included in the 
imputation models. From the Common Core of Data, significant predictors of tnissingness were 
Title I status, school enrollment, the percentage of students eligible for free or reduced-price 
lunch, the percentage of minority students, the percentage of girls, and the percentage of students 
proficient on the state mathematics assessment. From the sample-specific school aggregates, 10 
predictors of missingness were significant (gender, limited English proficiency, racial/ethnic 
minority status, special education status, free or reduced-price lunch status, state mathematics 
score, pretest score, general mathematics posttest score, and algebra posttest score). 

Analyses of missing data patterns suggested that a number of measured student and school 
characteristics were associated with missingness in a predictable pattern. Based on these analyses, 
researchers assumed that the data were missing at random (Rubin 1976, 1987; Schafer and Graham 
2002). 59 This assumption implies that missing data depend on observed data and not unobserved 
data. The missing at random assumption states that once all the observed data (that is, all covariates 
and auxiliary variables) are included in the study, the patterns of missingness are completely random. 
Graham (2009) refers to this type of missing data as “conditionally missing at random” to capture the 
covariate/auxiliary variable predictions. Given the missing at random assumption, multiple 
imputation was used to adjust for the missing data in the N-AR sample. 

Multiple Imputation 

With the goal of obtaining accurate parameter estimates for the relationships of interest in the 
impact models, researchers used multiple imputation by chained equations (in Stata) to impute 
values for missing student data. This approach involves developing an imputation model, cycling 
through each of the variables with missing data, and imputing them conditional on all the 
variables without missing data. The process starts with the variable with the fewest number of 
missing values and continues until no missing data remain. This process is then repeated multiple 
times using the new dataset until imputations stabilize (that is, the order in which the variables 
are imputed no longer matters) and a single dataset with no missing data is created (Stuart et al. 
2009). This entire process is repeated to create 10 imputed datasets without missing data. 

Imputation Model for N-AR Sample 

The model specified to impute the missing N-AR data had the following characteristics: 

• It included all significant (at p < .10) student-level predictors of missingness in the 
logistic regression models. 

• It included all significant (at p < .10) student- level interaction terms that predicted 
missingness in the logistic regression models. 


59 However, the missing at random assumption is not testable, and it is not possible to test whether missing 
data depend on the values that are missing (for example, a student with low posttest scores may be more likely 
to be absent during posttesting). 
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• It included all significant (at p < .10) school-level predictors of missingness in the logistic 
regression models to account for the clustered nature of the data. 

• It imputed the data for treatment and control samples separately. 

In choosing the number of variables to be included in the imputation model, researchers erred on 
the side of inclusion. The general recommendation for imputation models is to use every 
available variable in the imputation model (Little and Raghunathan 2004), including the 
dependent variable (Little and Rubin 2002; Allison 2002). Following these recommendations and 
those recently published by the Institute of Education Sciences on handling missing data (Puma 
et al. 2009), researchers included all variables in the impact model and an array of auxiliary 
variables. 

Including interaction terms can increase the specificity of imputation models. Because 
interaction terms are nonlinear (the product of two variables already included in the imputation 
model), they introduce nonlinear relationships into the imputed data (Graham 2009). Researchers 
therefore created student-level interaction tenns and tested whether they were related to missing 
data in a logistic regression model that included the relevant main effects and condition. A total 
of 17 student-level interaction terms were related to missing data and included in the imputation 
models: 

• pretest x general mathematics posttest 

• algebra posttest x special education 

• general mathematics posttest x special education 

• coursetaking x general mathematics posttest 

• coursetaking x algebra posttest 

• gender x racial/ethnic minority status 

• limited English proficiency x racial/ethnic minority status 

• limited English proficiency x special education 

• limited English proficiency x eligibility for free or reduced-price lunch 

• racial/cthnic minority status x time on pretest 

• gender x time on general mathematics posttest 

• special education x time on pretest 

• special education x time on general mathematics posttest 

• eligibility for free or reduced-price lunch x time on pretest 

• state mathematics score x time on general mathematics posttest 

• time on pretest x time on general mathematics posttest 

• time on pretest x time on algebra posttest. 60 


60 A model with a number of school-level interaction terms did not fit the data as well. 
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An important consideration when imputing data from cluster randomized trials in school settings 
is the multilevel structure of the data. To account for the clustered data, the study team included 
the school-level covariates and auxiliary variables described above that were significant 
predictors of student-level missing data. This approach allowed the imputation model to benefit 
from the relationships between school contextual factors and missing student data that are 
present after controlling for student-level auxiliary variables. 61 

Following recommendations by the Institute of Education Sciences (Puma et al. 2009), 
researchers performed the imputation on treatments and control schools separately. 

The final N-AR imputation model included 5 1 variables: 

• 3 outcome variables (student level) 

• 10 impact or sensitivity model covariates (5 student level, 5 school level) 

• 7 student-level auxiliary variables 

• 14 school-level auxiliary variables 

• 17 student-level interaction terms 

Imputation Model for Algebra-Ready Sample 

Because there was not sufficient statistical power to predict patterns of missingness in the AR sample 
given the limited number of missing data points (l%-3%), the imputation model developed for the 
N-AR sample was applied to the AR sample. Two additional student-level interactions (general 
mathematics posttest x algebra posttest, gender x time on pretest) that predicted missingness despite 
limited statistical power were included in the imputation model, as were six additional variables 
pertaining to high school coursetaking: 

• Final high school coursetaking outcome variable 

• Three variables used to code the final outcome (grade 9 course, perfonnance in grade 9 
course, planned grade 10 course). 

• Planned grade 9 course. 

• Indicator for whether students doubled up on full year mathematics courses in grade 9 or 10. 


61 Researchers also explored a fixed-effects approach using 63 dummy variables for schools to address the nested data. 
This model did not converge. Possible specification problems included the large number of dummy variables 
necessary to capture school differences (models with more than 35 dummy variables are more difficult to use 
[Graham 2009]). Another potential problem was the fact that the number of students nested within each school 
varied considerably. When including school-level dummy variables, low-incidence binary variables (such as special 
education status and limited English proficiency variables in this sample) will be constants within schools. Graham 
(2009) suggests performing a principal components analysis and examining Eigen values to determine the suitability 
of using dummy variables with low-incidence binary variables in the model. When this approach was conducted 
with the study data, the principal components analysis would not converge, suggesting that the dummy variable 
approach was inappropriate for the current study. 
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The final AR imputation model included 59 variables: 

• 9 outcome variables, including 6 additional variables used to code the high school 
coursetaking outcome (student level) 

• 10 impact or sensitivity model covariates (5 student level, 5 school level) 

• 7 student-level auxiliary variables 

• 14 school-level auxiliary variables 

• 19 student-level interaction terms. 

Graphic and Numeric Diagnostics 

Graphic diagnostics compare the distribution of observed values, imputed values, and combined 
observed and imputed values through kernel density plot estimations. “Imputed values” include 
only those values from each of the 10 datasets that were actually imputed; “observed and 
imputed values” refer to all the combined imputed and observed values from all 10 complete 
datasets. Graphic diagnostics can flag a potential misspecification of the multiple imputation 
model if, for example, the distribution of a particular posttest skewed to the right because 
students missing the posttest would not be expected to have higher imputed posttest values than 
students without the missing posttest. 

Diagnostics of imputed data do not provide a definitive test of the success of the imputation but 
rather a means to confirm that the imputed data do not seem overtly unreasonable. Differences 
between observed and imputed data do not necessarily mean that the imputation model 
specifications were incorrect. Substantive knowledge must be used in conjunction with the 
diagnostics when checking for potential misspecifications of the multiple imputation model 
(Stuart et al. 2009). For example, in this study, because students eligible for free or reduced-price 
lunch, students receiving special education services, and students with lower pretest and outcome 
scores were more likely to have missing data, it is reasonable to observe somewhat different 
distributions of observed and imputed posttest data. 

However, because the rates of missing data were relatively low (less than 10% on all variables 
except algebra posttest for non-algebra ready, on which 14% of cases were missing; less than 
3% on all variables for AR students), the best guide by which to judge the reasonableness of the 
distribution of imputed data is the observed data. Therefore, researchers examined whether the 
imputed data generated a similar distribution. The imputed data for the AR sample represent a 
small number of students (six for algebra, four for general mathematics, five for the study- 
administered pretest, and fewer than four for state mathematics scores). The distributions of 
imputed data for so few students (less than 3%) are less likely to follow the distribution of 
observed data in the AR sample (based on 434-437 students). 

Figure F-l shows that for the pretest, state mathematics scores (standardized), general 
mathematics posttest, and algebra posttest, the imputed and observed distributions are similar to 
the observed data. As an additional check on the multiple imputation model specification, 
researchers separately examined the density plot estimations of each of the individual 10 datasets 
(not presented here). Although there was variation among the 10 datasets, they did not detect any 
notable deviations from the expected distribution. 
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Numeric diagnostics analyzed the mean and the standard deviation of observed values, imputed 
values, and combined observed and imputed values in search of unreasonably large differences. 
Table F-6 presents these means and standard deviations for the pretest, state mathematics scores, 
and algebra and general mathematics posttests for the imputed and nonimputed data. Table F-7 
shows the percentage of students in each category of planned grade 9 courses for N-AR students, 
as well as planned grade 9 courses, actual high school coursetaking, and doubling up in grade 9 
or 10 by AR students. The descriptives parallel the graphic diagnostics, with very similar means 
and standard deviations and distributions of imputed and observed scores, suggesting a robust 
model of imputation. The tables also present the ratio of the difference between the mean of 
imputed and observed values to the standard deviation of observed values. For this ratio, Stuart 
et al. (1999) suggest that an absolute value greater than 2 may indicate that the variable should be 
flagged for further investigation. None of the ratios from the imputed datasets approached this 
threshold, and the differences in means and standard deviations were within reason. 

Figure F-l. Graphic Diagnostics for Pretest, State Math, General Mathematics Posttest, 
and Algebra Posttest 

a. N-AR Students 


Pretest 



General Math 



State Math (Standardized) 



Observed Values Only 

Observed and Imputed Values 

Imputed Values Only 

kernel = epanechnikov, bandwidth = 0.2000 


Algebra 
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Density Density 

.005 .01 .015 .02 0 .005 .01 .015 


b. AR Students 


Pretest 



Observed Values Only 

Observed and Imputed Values 

Imputed Values Only 

kernel = epanechnikov, bandwidth = 10.0000 


General Math 



State Math (Standardized) 



Observed Values Only 

Observed and Imputed Values 

Imputed Values Only 

kernel = epanechnikov, bandwidth = 0.2000 



AR is algebra ready. Non-AR is not algebra ready. 
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Table F-6. Numeric Diagnostics for Achievement Tests: Posttests, Pretest, and Prior 
Mathematics Scores 








Mean/ 

standard 

Standard 




Standard 



deviation 

deviation 

Type of student/test 

Number 

Mean 

deviation 

Minimum 

Maximum 

ratio 3 

ratio b 

N- AR 








General mathematics posttest 








Observed values 

1,342 

325.56 

28.94 

241 

399 

-0.41 

1.06 

Imputed values 

103 

313.61 

30.73 

241 

399 



Observed and imputed 
Algebra posttest 

1,445 

324.71 

29.23 

241 

399 



Observed values 

1,241 

429.60 

15.29 

400 

480 

-0.25 

1.07 

Imputed values 

204 

425.72 

16.30 

400 

480 



Observed and imputed 

1,445 

429.05 

15.50 

400 

480 



Pretest Promise Assessment 








Observed values 

1,384 

312.60 

27.22 

201 

392 

-0.69 

1.16 

Imputed values 

61 

293.91 

31.61 

201 

388 



Observed and imputed 

1,445 

311.81 

27.67 

201 

392 



State mathematics 








Observed values 

1,403 

-0.27 

0.90 

-2.36 

5.95 

-0.63 

1.20 

Imputed values 

42 

-0.30 

1.08 

-2.36 

5.01 



Observed and imputed 

1,445 

-0.28 

0.91 

-2.36 

5.95 



Algebra ready 

General mathematics posttest 
Observed values 

436 

361.57 

25.14 

267 

399 

-0.01 

1.14 

Imputed values 

4 

361.39 

28.75 

289 

399 



Observed and imputed 
Algebra posttest 

440 

361.56 

25.17 

267 

399 



Observed values 

434 

444.75 

14.10 

400 

490 

-0.07 

0.90 

Imputed values 

6 

443.83 

12.63 

420 

470 



Observed and imputed 

440 

444.73 

14.08 

400 

490 



Pretest Promise assessment 








Observed values 

435 

349.87 

23.25 

260 

399 

-0.22 

1.35 

Imputed values 

5 

344.68 

31.41 

260 

399 



Observed and imputed 

440 

349.81 

23.36 

260 

399 



State mathematics 








Observed values 

437 

0.90 

0.72 

-2.36 

2.32 

1.45 

1.53 

Imputed values 

3 

0.94 

1.10 

-2.17 

2.32 



Observed and imputed 

440 

0.90 

0.72 

-2.36 

2.32 




AR is algebra ready. Non-AR is not algebra ready. 

Note: Means and standard deviations are estimated across 10 imputed datasets. 

a. Ratio of the difference between the mean of imputed and observed values to the standard deviation of observed values. 

b. Ratio of the standard deviation of the imputed values to the standard deviation of the observed values. 

Source. Study records. 
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Table F-7. Numeric Diagnostics for Binary Coursetaking Outcomes 



Not intermediate 

Intermediate 

Variable 

Number 

Percent 

Number 

Percent 

N-AR students 

Planned coursetaking 





Observed values 

3,720 

28 

9,770 

72 

Imputed values 

437 

46 

523 

54 

Observed and imputed 

4,157 

29 

10,293 

71 


Not advanced 

Advanced 


Number 

Percent 

Number 

Percent 

AR students 

Planned coursetaking 





Observed values 

2,520 

59 

1,740 

41 

Imputed values 

77 

55 

63 

45 

Observed and imputed 

2,597 

59 

1,803 

41 

High school coursetaking 





Observed values 

2,510 

59 

1,760 

41 

Imputed values 

73 

56 

57 

44 

Observed and imputed 

2,583 

59 

1,817 

41 


Did not double up 

Doubled up 


Number 

Percent 

Number 

Percent 

Observed values 

3,340 

79 

890 

21 

Imputed values 

99 

58 

71 

42 

Observed and imputed 

3,439 

78 

961 

22 

AR is algebra ready. N-AR is 

not algebra ready. 





Note: Numbers represent total number of students across 10 imputed datasets. ForN-AR students, 0 = 
planned course below Algebra 1,1= planned course at or above Algebra 1 . For AR students, 0 = 
planned course at or below Algebra 1,1= planned course above Algebra 1. 

Source. Study records. 
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